Skip to content

Split traces larger than -b instead of dropping them - #170

Merged
adsr merged 1 commit into
masterfrom
chunked-traces
Oct 15, 2026
Merged

adsr merged 1 commit into
masterfrom
chunked-traces

Conversation

@adsr

@adsr adsr commented Oct 6, 2026

Copy link
Copy Markdown
Owner

Supersedes #167, which in turn superseded #157.

Currently phpspy drops traces that are larger than -b, which defaults to PIPE_BUF (4096 bytes). The idea is to avoid interleaved traces in -P mode. The problem is that traces are often larger than 4k in production, so we often bump -b and eat the cost of potentially corrupt traces.

The fix is to split large traces into smaller chunks. As long as each thread outputs each chunk under PIPE_BUF bytes, then readers like stackcollapse-phpspy.pl can reassemble the chunks back into the full trace.

Differences compared to #167:

  • Writes happen in a single pass. 167 did one pass to count chunks and another pass to write each chunk as k/n. Here we include a sentinel with each chunk that indicates "incomplete" (~) or "complete" (!).

  • There's no fixed reserve for trace_id records. It's calculated dynamically.

  • Traces under -b are unchanged -- no trace_id metadata.

  • No partial records. If a single frame record exceeds -b, drop it and emit a # truncated = 1 metadata record. We could make it a flag to emit partial records.

  • -1 mode is not a special case. 167 disabled chunking in -1 mode.

  • The -o fd is open-d then dup-d by other threads under a mutex. This avoids interleaving/corruption in -P -o ... mode. Adds a comment TODO the same for -O and -E. Could be a separate commit.

  • Delimiter stripping uses memchr -- more efficient than the existing byte loop. Could be a separate commit.

Supersedes #167, which in turn superseded #157.

Currently phpspy drops traces that are larger than `-b`, which
defaults to `PIPE_BUF` (4096 bytes). The idea is to avoid interleaved
traces in `-P` mode. The problem is that traces are often larger than
4k in production, so we often bump `-b` and eat the cost of
potentially corrupt traces.

The fix is to split large traces into smaller chunks. As long as each
thread outputs each chunk under `PIPE_BUF` bytes, then readers like
`stackcollapse-phpspy.pl` can reassemble the chunks back into the
full trace.

Differences compared to #167:

* Writes happen in a single pass. #167 did one pass to count chunks
  and another pass to write each chunk as `k/n`. Here we include a
  sentinel with each chunk that indicates "incomplete" (`~`) or
  "complete" (`!`).

* There's no fixed reserve for `trace_id` records. It's calculated
  dynamically.

* Traces under `-b` are unchanged -- no `trace_id` metadata.

* No partial records. If a single record exceeds `-b`, drop it and
  end the trace with a `# truncated = 1` chunk by itself.

* `-1` mode is not a special case. #167 disabled chunking in `-1`
  mode.

* The `-o` fd is `open`-d then `dup`-d by other threads under a
  mutex. This avoids interleaving/corruption in `-P -o ...` mode.
  Adds a comment TODO the same for `-O` and `-E`. Could be a separate
  commit.

* Delimiter stripping uses `memchr` -- more efficient than the
  existing byte loop. Could be a separate commit.
@adsr
adsr merged commit ffce945 into master Oct 15, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant