fix(runtime): give Pingora workers an 8 MiB stack and measure what uses it

[rickcrawford]

What this is

main is aborting. edf8b937 (#1235) is the commit that tipped it, but the AI request path was
already using more than half a Pingora worker's 2 MiB stack before that landed, so #1235 is the
straw and not the load. This raises the stack, and then measures what actually sits on it so the
next straw is visible before it lands rather than after.

thread 'Pingora HTTP Proxy Service' has overflowed its stack
fatal runtime error: stack overflow, aborting

That is the whole diagnostic. It does not unwind, it carries no backtrace, and it names no frame.

Part 1: an 8 MiB worker stack

Tokio gives a worker 2 MiB. Pingora never set thread_stack_size, so that default applied to
every worker sbproxy runs. The fix is in our fork
(soapbucket/pingora#2); this PR bumps the pinned
rev to pick it up and adds SB_WORKER_STACK_BYTES beside the existing SB_WORKER_THREADS.

The number, and the arithmetic

Measured:

build	measurement
macOS dev, fixture path with no TLS, guardrails, translator or holdback	survives 1,056 KiB, overflows 1,024 KiB
Linux dev, full request path	overflows the 2,048 KiB default

Half the old budget was already spent on a stripped-down path, and the real one exceeds all of
it. Both of the old probes broke at the same boundary, which says the cost belongs to the shared
dispatch chain rather than to any one branch of it.

Shrinking the path is not an available answer. Two future extractions on this codebase moved 528
bytes against a 2,097,152-byte stack: 0.025 percent, for an afternoon. Recovering a real margin
that way needs about two thousand of them.

4 MiB is not an answer either. It puts us past half again on day one, which is the position that
produced this outage, and buys one release.

8 MiB is the RLIMIT_STACK default Linux gives a process's main thread, so it is the size the
platform already treats as normal for a thread running arbitrary code, and it is four times
tokio's worker default.

What it costs, for the worker count sbproxy actually runs (threads = available_parallelism(), one proxy service, plus pingora's 1-thread server runtime, so roughly
17 to 33 threads on a 16-vCPU host):

stack	17 threads	33 threads
2 MiB	34 MiB	66 MiB
8 MiB	136 MiB	264 MiB

That is reserved address space, out of the 128 TiB a 64-bit process has, and it is 0.0002 percent
of it. Resident memory does not move: a thread stack is an anonymous mapping the kernel commits
page by page on first touch, so RSS tracks the depth a request reaches, which this does not
change.

The reason it stops at 8 and not 16: thread_stack_size reaches the blocking pool too, whose
tokio default cap is 512 threads. Worst-case reservation per runtime is 4 GiB at 8 MiB and 8 GiB
at 16 MiB. Both are VmSize and not VmRSS, but 8 GiB is a number an operator stops and asks
about, and some container runtimes cap address space outright.

If 8 turns out to be wrong, SB_WORKER_STACK_BYTES and pingora's runtime_thread_stack_size are
both one value away, with no fork change.

Part 2: measuring what uses the stack

Raising the ceiling without a floor gauge trades a painful signal for no signal.

Why the old guards could not see this

They measured core::mem::size_of on a single future. A future's size is the state it holds
between polls. The stack is the whole chain of frames live during one, and these futures sit
at the top of that chain rather than containing it. On the tree that was aborting, they reported:

guard	measured	budget	used
request_filter_fits_its_share_of_a_pingora_worker_stack	36,480	262,144	13.9%
the_dispatch_future_has_not_grown	24,464	32,656	74.9% of 1.2% of the stack

Roughly 96 percent of what fills the stack was outside what either could see, and both stayed
green through three overflows.

The numbers

Measured with the new probe, macOS arm64, dev profile, driving request_filter on a real
Pingora worker:

tree	bytes of worker stack	of the old 2 MiB	of the new 8 MiB
c84feb1c (#1236, last green)	1,569,776	74.8%	18.7%
edf8b937 (#1235, broke main)	1,569,776	74.8%	18.7%
this branch	1,600,928	76.3%	19.1%

The two adjacent commits measure identically, and that is worth stating plainly rather than
hiding. edf8b937 adds one RequestContext field on this path, about 64 bytes, which lands
in existing debug frame slack; its other 590 lines sit behind authentication: type: hmac_auth,
which this fixture does not configure. The measurement does resolve growth (+31,152 bytes
between edf8b937 and 16b94ea2), it just has nothing to resolve across that particular pair.

Which is the real finding: at 74.8 percent of the old budget before #1235, the commit that
tips it over does not have to be large, and looking for it in the diff was never going to work.

Read the number as the depth of the configuration it drives, which is a floor. The e2e origin
that actually aborts (compression-cel.localhost) wires compression, a CEL policy plane and
guardrails. This check holds the floor with a number on every PR; the smoke lane, now
unfiltered, holds the ceiling with a real binary.

Red-first

Since the calibration came back null, here is the proof the number moves, aimed at the exact
function #1235 grew. One local added to request_phase::request_filter, same commit, same
fixture:

	STACK_HIGH_WATER_BYTES
c84feb1c	1,569,776
c84feb1c + let red_first_pad = [0u8; 96 * 1024]	1,766,672
delta	+196,896 for 98,304 bytes added

Roughly twice the local, which is expected: request_filter is an async fn, so a local held
across an .await lives in the future's state and in the frame the future is polled in, and a
debug build keeps both.

The budget, set from the Linux measurement

2,359,296, which is ceil(1_604_072 / 256 KiB) * 256 KiB + 512 KiB, where 1,604,072 is the
STACK_HIGH_WATER_BYTES this PR's own production request-path smoke lane printed on Linux.

tree	streaming	buffered
Linux, from the smoke lane	1,604,072	1,471,496
macOS, aarch64, local	1,600,952	1,468,456
delta	+3,120	+3,040

An earlier draft of this used half the worker stack, 4,194,304, which was 2.6x of slack against
a measured 1.6 MiB and would not have noticed fifty commits the size of the one that broke
main. It measured the right quantity and held it to nothing.

The delta above also corrects a claim I made repeatedly while building this: that Linux debug
frames are much larger and macOS numbers only rank candidates. For this path they agree to
within 0.2 percent. Linux is not why the smoke lane overflowed 2 MiB while this fixture sits at
1.6 MiB; the configuration that lane runs is, and the fixture wires none of it. The margin for
that lives in the 8 MiB stack, not in this budget.

What replaces them

pingora_runtime::worker_stack records the address of a local in each worker thread's entry
frame. crates/sbproxy-core/src/server/stack_probe.rs takes the address of a local at the
deepest point of the request path and reports the difference: bytes actually in use.

the_ai_dispatch_path_stays_inside_its_stack_budget drives a whole streamed AI request through
request_phase::request_filter, on a real Pingora worker whose stack is set to exactly the
budget in scripts/stack-budget-baseline.count. Entering at request_filter rather than at
handle_ai_proxy is the point: request_filter stays live while everything it dispatches into
runs above it, and #1235 grew request_filter, proxy_http, context, trust_tier and
hmac_auth while touching no AI file at all. A measurement that started lower would not have
seen a byte of it.

Two things are checked, and the second matters as much as the first:

1. The request completes. A path that outgrows the budget overflows, which under nextest fails
   this test and nothing else, and that covers the entire chain including every frame below the
   probes: reqwest, hyper, serde, the TLS stack.
2. The probe reported a non-zero depth. Without this the budget check could pass vacuously if the
   measurement ever stopped working, which is exactly the failure mode this whole change exists
   to end.

Cost on the request path

Release builds: one call, two thread-local reads, a subtraction, a comparison, and a branch
that is not taken once a worker has settled. The process-wide maximum is written only when a
thread beats its own record, so the steady state touches no shared memory and no lock. Probes
fire once per request, not once per chunk: a loop body's frame is one size on every iteration,
so a second measurement learns nothing.

Debug builds cost more than that, and my earlier claim did not say so. A StackMark carries
the ThreadId that made it so used_here can refuse a cross-thread comparison, which means
each probe also makes two std::thread::current().id() calls, and thread::current() clones an
Arc, so that is two atomic increments and two decrements per probe. Two probes per request.
It is gone entirely under cfg(not(debug_assertions)), so production is unaffected, but "costs
effectively nothing" was only ever true of the build that is not the one CI runs.

No unsafe. Taking the address of a local, casting a reference to a raw pointer and casting that
pointer to an integer are all safe; only a dereference would not be, and nothing here
dereferences. Neither constraint was traded.

What it cannot see, stated plainly: the reported number is the depth at the probes, not the
deepest point reached, so read it as a floor and a trend. The budget in (1) is the part that
covers everything.

The operator surface, and what it cannot see

publish warns once per process, when a request first passes three quarters of the worker stack,
with the bytes used and the stack size. Nothing has failed at that point, but a stack overflow
leaves no diagnosis, so this is the last chance to say anything at all. Documented in
docs/manual.md beside the knob that answers it.

It can only fire for AI traffic. Both probes are on the AI dispatch path: one in the
streaming relay's loop, one at the buffered relay's terminal write. A deployment that proxies
plain HTTP and never dispatches to a provider has no probe on its request path, so its
high-water stays zero and the warning cannot fire no matter how deep that path goes. That is the
right first place to instrument, because the AI path is the deep one and the one that aborted,
but the warning is not a general stack-safety signal for the proxy and should not be read as
one.

Part 3: the lane that did not run

The smoke lane lived inside context-compression-eval.yml behind a twelve-path trigger. Neither
commit since it last passed touched a trigger path, which is why main sat broken with nothing
to say so. The lane did not go red; it did not run.

A stack budget cannot be gated on a path list. The stack is consumed by the whole call chain, so
a frame added in a crate that trigger list has never heard of spends exactly as much as one added
in ai_dispatch.rs. Any filter narrower than "every pull request" is narrower than the thing
being guarded.

request-path-smoke.yml is that job moved into its own workflow with no path filter, keeping the
check name production request-path smoke. It gains a first step that runs the stack budget test
with --nocapture, so the Linux number is printed in the lane that broke, ahead of the three e2e
runs it would otherwise fail behind.

Ratchet

scripts/check-stack-budget-ratchet.sh, the fourth of the family and the same shape as the other
three: one integer in one file, compared against the merge base, allowed to fall and not to rise.
It is wired into check.sh, check-fast.sh, and the ci.yml test lane.

The budget is half of DEFAULT_THREAD_STACK_SIZE. The invariant is "the request path fits in half
a worker's stack", which stays meaningful if the worker stack is ever changed again and leaves the
other half for the TLS, guardrail, translator and holdback frames the fixture does not wire.

post-merge-rederive.sh prints it rather than recomputing it, with the merge rule written down:
this number has no scanner behind it, so on a merge the correct value is the lower of the two
sides, never their maximum.

What the fork review changed here

Fork PR soapbucket/pingora#2 came back MERGE with findings; the one that reaches this repo is
M1. Pingora's offload pools spawned plain std threads with neither the configured stack nor
a stack base, and sbproxy runs two of them: upstream_connect_offload_threadpools: Some(2) at
crates/sbproxy-core/src/server/lifecycle.rs, plus the downstream TLS handshake pool inside
pingora's rustls listener. So upstream DNS and connect, and the TLS handshake, were still on
the platform default stack while the workers had four times as much, and the instrument could
not see either of them.

The fork now carries a process-wide default that Server::run sets from
runtime_thread_stack_size before anything spawns, and the offload pools read it. sbproxy needs
no change for this: leaving runtime_thread_stack_size unset resolves to 8 MiB for workers,
blocking pools and offload pools alike, and SB_WORKER_STACK_BYTES moves all of them together.

The one API change that reaches this diff: worker_stack::here() returns a StackMark instead
of a bare usize, carrying in debug builds the ThreadId that made it so used_here can refuse
a cross-thread comparison. Release builds carry neither the field nor the check, so the hot-path
cost is unchanged.

Fork provenance, and a wrong instruction in the release procedure

Riding with the lockfile re-pin because it is the same subject.

The comment above [patch.crates-io] now records what the fork actually is. The crates declare
0.8.0 and the branch is named for it, and neither is a release we track: the tree carries 167
upstream commits the 0.8.0 tag does not, plus our seven. Read the version as an API generation,
not as something to match an advisory against. The concrete numbers come from
scripts/divergence.sh in the fork, which its CI prints on every PR:

base:   0046038 (2026-08-07)
ahead:  7 commits of ours
behind: 23 upstream commits
files:  20 differ, 14 outside .github/

CLAUDE.md and AGENTS.md both told the release procedure to rebase sbproxy-0.8.0 onto "the
target upstream tag". That is the step somebody actually follows at a release, and following it
would move the fork onto a different line. Cloudflare cuts releases on a release branch, so
0.8.1 is not an ancestor of main: it holds 8 commits main lacks while main holds 190 it
lacks (git rev-list --count origin/main..0.8.1 and the reverse). It would also strand the
current_handle() fix upstreamed at cloudflare/pingora#982, which lands on main.

Both files now say to rebase onto a newer upstream main, give the release-branch reason in a
sentence so nobody re-derives it, point at scripts/divergence.sh, and carry the lockfile rule
this branch followed. Fixing only CLAUDE.md would have left the identical wrong instruction in
the file half the tooling reads.

Lockfile

branch = "sbproxy-0.8.0" is unchanged; only the pinned rev moves, to 2d9dc2b, the merged
fork tip. The entire diff against main:

    11 < source = "git+.../pingora?branch=sbproxy-0.8.0#4153f0d22ef5ccc591a3ae5e75637effe83f9be8"
    11 > source = "git+.../pingora?branch=sbproxy-0.8.0#2d9dc2b908dd1dee69ade3378054c0d8b472bac5"
     1 >  "pingora-runtime",

Eleven rev bumps, one per patched crate, and one dependency edge for worker_stack. Nothing
else: no windows-sys, no itertools, no version changes anywhere. Produced with cargo metadata --offline rather than cargo update -p, and read line by line rather than by stat,
because a cargo update -p on this workspace has silently downgraded unrelated dependencies
before.

Adversarial review

Reviewer: independent review agent against .github/code-review-rubric.md, verdict NOT READY, report at docs/sbproxy/loop-2026-08-28/review-pingora-stack-sbproxy.md
Findings: 1 Blocker, 6 Major, 5 Minor
Verification: both stack budget tests re-run with --nocapture and their numbers read out of the log (streaming 1,600,952, buffered 1,468,456, budget 2,621,440, worker stack 8,388,608); the ratchet's fail-closed paths exercised in a scratch repository with no main and no origin; the same script run in CI in the shallow-checkout job that used to skip it, to watch it go red there rather than only on a laptop; full scripts/check.sh re-run after every fix.

• Blocker - scripts/check-stack-budget-ratchet.sh:150 - resolved its base with git merge-base HEAD origin/main while wired into ci.yml's lint job, whose checkout has no fetch-depth, so on a pull_request event no base resolved and it exited 0: it ran on every pull request and checked nothing. Fixed in a70136c87: fails closed, takes STACK_BUDGET_BASE_REF the way the changelog guard does, and moved to the guards job whose checkout is full depth.
• Major - crates/sbproxy-core/src/server/ai_dispatch.rs:23445 - the worker was built .thread_stack_size(STACK_BUDGET), so used <= budget could not fail without the process having already aborted, and the abort sat at half the production stack. Fixed in a70136c87: the worker is sized to DEFAULT_THREAD_STACK_SIZE and the budget is the assertion.
• Major - scripts/stack-budget-baseline.count:1 - 4,194,304 against 1,600,952 measured is 2.6x of slack, and the commit that broke main moved the depth 31,152 bytes. Fixed in a70136c87: set from the measurement.
• Major - docs/manual.md:2355 - SB_WORKER_STACK_BYTES missing from the environment-variable table, and the count still said fourteen. Fixed in a70136c87.
• Major - crates/sbproxy-core/src/server/lifecycle.rs:3625 - the sub-page refusal documented in docs/manual.md never ran, because ServerConf::validate() is called only from from_yaml and sbproxy passes a struct literal, so SB_WORKER_STACK_BYTES=8 started and aborted on the first request. Fixed in a70136c87: resolve_worker_stack_bytes refuses it and the manual says what actually happens.
• Major - crates/sbproxy-core/src/server/lifecycle.rs:3612 - the whole hunk that reads the knob had no test and could be reverted with every test still passing. Fixed in a70136c87: worker_stack_env_tests, four cases.
• Major - crates/sbproxy-core/src/server/request_phase.rs:1035 - five comments still described the worker stack as 2 MiB, one of them inside an assertion message an operator reads. Fixed in a70136c87; two further references left at 2 MiB because they are historical.
• Minor - crates/sbproxy-core/src/server/ai_dispatch.rs:16446 - deleting a_non_streaming_dispatch_fits_a_pingora_worker_stack left the buffered relay with no stack coverage and its probe unreached by any test. Fixed in a70136c87: the_buffered_ai_dispatch_path_stays_inside_its_stack_budget.
• Minor - .github/workflows/ci.yml:358 - the ratchet's --self-test was never invoked while its three siblings all run. Fixed in a70136c87.
• Minor - .github/workflows/request-path-smoke.yml:69 - filtered by substring, and cargo test exits 0 when a filter matches nothing, so a renamed test would make the step pass while checking nothing. Fixed in a70136c87: the step counts the tests that ran and fails if it is not exactly two.
• Minor - .github/workflows/request-path-smoke.yml:24 - unscoped pull_request: trigger. Fixed in a70136c87: scoped to branches: [main].
• Minor - docs/.changes/20260828-pingora-worker-threads-now-get-an.json:1 - fragment typed fixed while introducing a knob. Fixed in a70136c87: retyped changed.