Skip to content

Scale the cpu-diff regression gate floor with the workload - #212

Merged
jantimon merged 1 commit into
mainfrom
fix/cpu-diff-noise-floor
Aug 2, 2026
Merged

Scale the cpu-diff regression gate floor with the workload#212
jantimon merged 1 commit into
mainfrom
fix/cpu-diff-noise-floor

Conversation

@jantimon

@jantimon jantimon commented Aug 2, 2026

Copy link
Copy Markdown
Owner

Problem

cpu-diff --fail-on-regression is the JS-cost gate the docs recommend, but its noise floor was a fixed 0.5 ms on the net JS self-time. That floor false-reds byte-identical code, and it gets worse with --iterations: summed self-time grows while the floor stays absolute, so a larger (or higher-iteration) workload trips constantly on sampling bursts.

Measured (all --target node, gate lock held, one measurement at a time)

Run-to-run net JS-self delta on byte-identical code, and the fixed-0.5 ms false-red rate:

workload fixed 0.5 ms false-red abs net noise (p95) rel net noise (p95)
~5 ms (18 samples) 2% 0.49 ms 10.8%
~44 ms 27% 1.54 ms 3.4%
~220 ms (--iterations 20) 40% 6.27 ms 2.8%

Absolute jitter grows with self-time; relative jitter shrinks (finer quantization). A single absolute floor cannot fit both ends. This reproduces the integrator report exactly (red ~1 run in 4 at ~10 ms, worse with iterations).

Fix

The gate trips only when the net clears max(--noise-floor ms, --noise-pct% of the baseline), default max(0.5 ms, 15%). The percentage term tracks the absolute jitter as it grows; the absolute term keeps a small or 0 ms baseline from gating on a fraction of a sample. 15% sits above the worst measured identical-pair relative jitter (~11% at 5 ms) with margin.

New flags --noise-floor <ms> / --noise-pct <n> widen (noisier host) or tighten (large stable workload) each term. CpuDiffResult JSON now carries noisePct and the effective gateFloorMs, so a consumer reproduces the exit code from JSON.

Verified after the fix (built CLI, real recordings, actual exit codes)

  • Identical pairs, default gate: 66/66 green at ~5 ms, 45/45 at ~220 ms, 28/28 on a same-path base-vs-base (100% green).
  • Real +30% regression on a 5 ms workload (same module path, content changed +30%, so no comparability refusal): 64/64 caught across an 8×8 base-vs-current cross-product. A single pair misses only when the baseline recording itself over-samples high (a baseline-quality artifact the committed-once baseline avoids), not the floor being loose.
  • The resolving floor (both sides below ~2 ms = not evaluable) still takes precedence; comparability refusals unchanged.

Scope

  • src/commands/cpudiff.ts: the two-term gate floor + flag plumbing.
  • src/cli.ts: --noise-floor / --noise-pct (validated by the existing numeric policy).
  • src/model/query.ts: CpuDiffResult gains noisePct + gateFloorMs.
  • Docs: cpu-profiling.md noise-floor section rewritten present-tense; facts.md row updated + a scaling-floor row added; README gate paragraph.
  • Tests: 6 new gate-floor unit tests; measurement-lane comment refreshed.

Gates: lint, format:check, build, unit (744), measurement (6) all green. knip is unaffected by this change (it flags only worktree-local devDependency resolution here).

The cpu-diff --fail-on-regression gate tripped on byte-identical code against
a fixed 0.5ms net floor, and worsened with --iterations: summed self-time grows
while the floor stays absolute. Measured on --target node, the run-to-run net
delta on identical code is ~0.5ms/~11% at a 5ms workload and ~6ms/~3% at 220ms;
a 0.5ms floor false-reds ~2% of identical pairs at 5ms but ~40% at 220ms.

Gate on max(--noise-floor ms, --noise-pct% of baseline), default max(0.5ms, 15%).
The percentage term tracks the absolute jitter as it grows, so identical code
gates green >=95% at any iteration count (66/66 at 5ms, 45/45 at 220ms) while a
30% regression on a 5ms workload still clears its 0.75ms floor (64/64 caught).
The absolute term keeps a small or 0ms baseline from gating on a fraction of a
sample. New flags --noise-floor/--noise-pct widen or tighten each term;
CpuDiffResult JSON carries noisePct and the effective gateFloorMs.
@jantimon
jantimon merged commit 1af338b into main Aug 2, 2026
4 checks passed
@jantimon
jantimon deleted the fix/cpu-diff-noise-floor branch August 2, 2026 15:19
@github-actions github-actions Bot mentioned this pull request Aug 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant