Benchmark the path a change actually touches

[deverman]

User outcome

A change that affects performance is measured on the path it changed, against a comparable baseline, and produces an explicit verdict: better, worse, or indistinguishable — with a stated merge recommendation. Nobody has to reconstruct that judgement by reading raw latency tables.

Validation impact

package — development and release tooling. No production code path changes.

Evidence: what went wrong on #88

#88 changed list_projects. Validating it consumed roughly two hours and four 10-minute smoke suites, and none of them measured the changed code.

1. The suite does not cover the changed tool. scripts/benchmark-suite.sh runs benchmark-list-tasks, benchmark-task-counts, and benchmark-project-counts. There is no list_projects benchmark, so a list_projects change is invisible to the gate that is nevertheless required to pass before merging it.

2. Pass/fail is decided by a single call on a host with 3-4x variance. The suite fails closed if any measured call fails. Observed across four runs on one machine:

   run	build	measured	failures
   1	branch	66	1
   2	branch	338	1
   3	master	315	0
   4	branch	265	0

   2-in-404 versus 0-in-315 is about p≈0.5 by Fisher's exact test — indistinguishable from chance. The gate flipped twice on noise.

3. Host variance dwarfs the effects being measured. Same code, consecutive runs: list_tasks p50 of 9259 ms and 11838 ms. Master's single run showed 2692 ms — roughly 4x below both branch runs, making the baseline the outlier. The variance did not track load average: the highest-load run had the shortest get_task_counts tail (14209 ms vs master's 29611 ms at lower load).

4. The deadline sits inside the noise band. Master's own passing run peaked at 29611 ms against a 45 s timeout — 66% of the deadline on a run with zero failures. A suite that fails closed on one stall, on a host whose tail routinely reaches two-thirds of the limit, will keep producing runs 1 and 2 indefinitely.

5. There is no numeric latency gate at all. scripts/check-benchmark-suite-results.sh checks success and coverage; it contains no latency threshold. So the suite can neither detect a real regression nor confirm an improvement — it only detects total failure.

What actually answered the question: a 10-minute interleaved A/B on list_projects itself — 10 samples per variant, alternating, same host, same session. It showed the unchanged path at 385 ms vs 390 ms on master (noise), an opt-in cost of +50 ms (+13%) when folder fields are requested, and the target workflow at identical latency with a quarter of the payload. That produced a defensible merge verdict in a fraction of the time.

Proposed scope

1. Measure the path that changed

• Map tools to benchmarks and fail loudly when a change touches a tool with no benchmark coverage, rather than passing a suite that never exercised it.
• Add list_projects, list_tags, and list_folders coverage, or provide a generic single-tool harness that can benchmark any tool by name.

2. Compare against a baseline, interleaved

• Support --baseline <ref>: build the comparison ref, run both, and interleave samples so host drift hits both arms equally. Sequential whole-suite comparisons are what produced the misleading 4x reading above.
• Record n, median, stdev, and standard error per arm — not just p50/p95/p99 of a single arm.

3. State when a result is better, worse, or unknown

Explicit, written-down criteria rather than judgement per PR. Suggested starting point, to be tuned once real variance data exists:

• Regression: median increase exceeding both 5% and 3 standard errors.
• Improvement: median decrease meeting the same bar.
• Indistinguishable: everything else — reported as such, never as a win.
• Inconclusive: stdev/median above a threshold (host too noisy), or n below the minimum. This must be a distinct outcome from "no change".
• Reliability judged by failure rate with a minimum sample size, not by a single failed call. A stalled call on a noisy host should register as one observation, not a verdict.

4. Emit a merge recommendation

The suite should end with a plain statement, for example:

VERDICT: indistinguishable (median +1.3%, 0.4 SE, n=10/arm, host stdev 7%)
COVERAGE: list_projects benchmarked (changed tool)
RELIABILITY: 0 failures / 265 calls (baseline 0 / 315)
RECOMMENDATION: safe to merge — no regression detected on any measured path

with INCONCLUSIVE - rerun on a quiescent host as a first-class outcome.

5. Make deadlines and host conditions explicit

• Derive the per-request deadline from observed p99 with headroom, or record the p99-to-deadline ratio so a run near the limit is visibly at risk rather than silently lucky.
• Capture host conditions (load, uptime, competing processes) into the artifact. Note that load average proved a poor predictor here, so record it as context, not as a gate.

Safety and compatibility

• Tooling only; no production behavior changes.
• The existing profiles (canary, smoke, release, stress) should keep their names and meanings.
• Related: Detect host sleep interruptions in benchmark evidence #193 tracks host sleep disturbing benchmark evidence — the same class of problem, and the two should be resolved consistently.

Acceptance criteria

• Changing a tool with no benchmark coverage fails the validation step with a clear message naming the uncovered tool.
• A baseline comparison can be run in one command and interleaves its samples.
• Output includes n, median, stdev, and standard error per arm.
• Every run ends with an explicit verdict and merge recommendation, including a distinct INCONCLUSIVE outcome.
• Reliability is expressed as a failure rate against a minimum sample size; one stalled call cannot by itself fail a run.
• The Expose project folder membership and root filtering in list_projects #88 scenario is reproducible end to end: a list_projects change produces a verdict without anyone hand-rolling an A/B.

Non-goals

• Rewriting the benchmark harness or changing what the realistic 1.5-hour release suite measures.
• Chasing the underlying host variance itself, which is an environment property rather than a product defect.
• Per-commit CI benchmarking.

---

Status update (2026-08-03)

Retitled to drop "and state a merge verdict". Everything above is preserved as
the original record — the evidence from #88 still stands and still motivates
the remaining work. Only the scope has narrowed.

Shipped (#211). Coverage detection: focusrelay-dev classify now warns when
a change touches a tool the benchmark suite cannot measure, naming the tool and
explaining why a green suite does not clear the change. This is the half that
was costing real time on #88.

Removed (#213). The BenchmarkVerdict engine — interleaved A/B comparison,
better/worse/inconclusive criteria, and the merge recommendation. It merged with
no caller: 130 lines of library and ten tests that nothing in the product
invoked. It was deleted rather than wired up, because no caller had asked for
it. The acceptance criteria above that describe verdicts, baseline comparison,
and per-arm statistics are therefore not currently in scope.

Remaining. Benchmarks for the six public tools the suite cannot measure.
Add them when a change to one of those tools actually needs measuring, not in
advance — see the standing decision "Do not build tooling ahead of a caller" in
docs/roadmap-execution-plan.md.

[deverman]

First application of these criteria: #171

#171 was benchmarked using the approach this issue proposes, before the tooling exists. Recording what worked and what still needs building.

What the approach produced

Interleaved A/B on the changed path, 6 reps per arm, 10 names, one MCP session per measurement:

arm                  n   median    stdev     SE   round trips
10 scalar searches   6   2374ms     44ms   18ms            10
one batch call       6    429ms     33ms   13ms             1

VERDICT: IMPROVEMENT — median −81.9% (−1945ms), 86.7 standard errors

An unambiguous verdict in about ten minutes, against a host whose whole-suite variance is 3-4x. Interleaving is what made that possible: both arms absorbed the same drift.

What this confirms about the proposal

1. Coverage checking is the highest-value piece. list_projects and list_tags still have no suite coverage, so Resolve multiple project or tag names in one bounded query #171 — like Expose project folder membership and root filtering in list_projects #88 — could have passed the required gate without being measured at all.
2. Standard error is the right yardstick, not raw percentage. 86.7 SE here is unarguable; Expose project folder membership and root filtering in list_projects #88's folder-field cost was +13% at roughly 5 SE, also real; the Expose project folder membership and root filtering in list_projects #88 smoke failures were ~0.4% at well under 1 SE and should never have read as signal.
3. Per-arm n, stdev, and SE belong in the output. Without stdev there is no way to tell 81.9% from noise on this host.

What the tooling still needs to add

• Correctness before performance. Checking that batch and scalar agreed for the same name found a real bug — a request for fields: ["id"] fetched nameless items and matched nothing, so batch returned 0 where scalar returned 2/1/3. Benchmarking a broken implementation would have produced a beautiful, meaningless improvement. Any A/B harness should require an equivalence assertion between arms before it reports timings.
• Report round trips and bytes alongside latency. Latency alone would have hidden that bytes rose (one grouped envelope replacing ten bare ones) while round trips fell 10 to 1. Both matter, and they can move in opposite directions.
• Sessions matter. Each measurement used one MCP session so process startup did not dominate; a harness should make that the default rather than something the operator remembers.

Suggested refinement to the verdict criteria

Add a fourth outcome beyond improvement / regression / indistinguishable / inconclusive: not comparable — when the arms do not return equivalent results, so timings are meaningless. That is exactly the state #171 was in before the field bug was fixed.

[deverman]

Roadmap status: parked after the useful slice shipped

#211 shipped the coverage warning that prevents a green benchmark from clearing an unmeasured changed tool. The remaining six benchmarks are intentionally unmilestoned and should be added only when a scheduled change actually touches one of those tools and needs measurement. This preserves the issue's own "do not build tooling ahead of a caller" decision.