Scale the cpu-diff regression gate floor with the workload - #212
Merged
Conversation
The cpu-diff --fail-on-regression gate tripped on byte-identical code against a fixed 0.5ms net floor, and worsened with --iterations: summed self-time grows while the floor stays absolute. Measured on --target node, the run-to-run net delta on identical code is ~0.5ms/~11% at a 5ms workload and ~6ms/~3% at 220ms; a 0.5ms floor false-reds ~2% of identical pairs at 5ms but ~40% at 220ms. Gate on max(--noise-floor ms, --noise-pct% of baseline), default max(0.5ms, 15%). The percentage term tracks the absolute jitter as it grows, so identical code gates green >=95% at any iteration count (66/66 at 5ms, 45/45 at 220ms) while a 30% regression on a 5ms workload still clears its 0.75ms floor (64/64 caught). The absolute term keeps a small or 0ms baseline from gating on a fraction of a sample. New flags --noise-floor/--noise-pct widen or tighten each term; CpuDiffResult JSON carries noisePct and the effective gateFloorMs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
cpu-diff --fail-on-regressionis the JS-cost gate the docs recommend, but its noise floor was a fixed 0.5 ms on the net JS self-time. That floor false-reds byte-identical code, and it gets worse with--iterations: summed self-time grows while the floor stays absolute, so a larger (or higher-iteration) workload trips constantly on sampling bursts.Measured (all
--target node, gate lock held, one measurement at a time)Run-to-run net JS-self delta on byte-identical code, and the fixed-0.5 ms false-red rate:
--iterations 20)Absolute jitter grows with self-time; relative jitter shrinks (finer quantization). A single absolute floor cannot fit both ends. This reproduces the integrator report exactly (red ~1 run in 4 at ~10 ms, worse with iterations).
Fix
The gate trips only when the net clears
max(--noise-floor ms, --noise-pct% of the baseline), defaultmax(0.5 ms, 15%). The percentage term tracks the absolute jitter as it grows; the absolute term keeps a small or 0 ms baseline from gating on a fraction of a sample. 15% sits above the worst measured identical-pair relative jitter (~11% at 5 ms) with margin.New flags
--noise-floor <ms>/--noise-pct <n>widen (noisier host) or tighten (large stable workload) each term.CpuDiffResultJSON now carriesnoisePctand the effectivegateFloorMs, so a consumer reproduces the exit code from JSON.Verified after the fix (built CLI, real recordings, actual exit codes)
Scope
src/commands/cpudiff.ts: the two-term gate floor + flag plumbing.src/cli.ts:--noise-floor/--noise-pct(validated by the existing numeric policy).src/model/query.ts:CpuDiffResultgainsnoisePct+gateFloorMs.Gates: lint, format:check, build, unit (744), measurement (6) all green. knip is unaffected by this change (it flags only worktree-local devDependency resolution here).