Prove a task can pass before buying it

[bryfa]

Refs bryfa/mulhacenlabs-work#1031

Every expensive thing measured on 2026-09-10 had the same shape: work was dispatched before anyone had proved it could succeed. Not five problems — one problem, five times. The controls that already exist (meter, budget, abort) all fire after the spend. This is the half that comes before.

Base is fix/configurable-check-timeout, the checkout every factory runs, so this stacks on the cost controls rather than conflicting with them. An upstream PR gets cut separately once NateBJones-Projects#74 and NateBJones-Projects#129 land.

1. Baseline stops being a flag

run --baseline shipped already and had never once been used here — which is what a flag people are supposed to remember is worth. Every run now baselines itself first.

A run is refused before any worker spawns (exit 2) on the two shapes that make a task unbuyable:

baseline says	meaning	dispatch
FAIL	the check demands behavior that does not exist yet	✅ wanted
pass	already green, so green at the end too — it cannot tell you the work happened	❌ refused
error	the check could not be executed at all	❌ refused

Refusing on a failing baseline would refuse every honest manifest, so it does not.

⚠️ Baseline does not prove a worker can write your deliverables, and no phase that spawns no workers could. It probes declared absolute expect_files as the dispatcher sees them, so it catches paths the filesystem itself refuses; a sandbox can still deny a path that looks writable from here. Three layers divide that ground:

layer	catches	how
lint (worker_unwritable_paths, already shipped in NateBJones-Projects#129)	the spec hands a worker an absolute path outside its own task dir — sandbox scope, and the shape of the $19.43 incident	statically
baseline (this PR)	a path no process could write, whoever asked	filesystem probe
canary (this PR)	whatever neither could know statically	bought once, not once per task

2. The canary is a stop, not a smaller batch

A multi-task run releases its first task alone, judges it by its own executed check, and only then releases the rest. A bad verdict marks every remaining task SKIPPED without spawning.

Measured both ways on the same night: a 3-ticket run that did this caught a design fault on its first task; a 32-task run that did not lost a whole round to a fault its first task had already demonstrated.

--canary-confirm adds a human on top — deliberately not default. A pause met on every run becomes a keypress people learn to hit, which buys the appearance of a gate and none of the substance. On a non-TTY it stops rather than fabricating approval.

3. Escapes that announce themselves

./ringer.py run swarm.json --no-baseline "checks assert unchanged invariants on purpose"
./ringer.py run swarm.json --no-canary   "tasks are fully independent"

Each takes a reason, prints WAIVED (not proved, not verified), and records it. A blank reason is rejected rather than silently re-enabling the gate the operator believed they had turned off.

Every run record carries a preflight block with the per-task baseline verdict and the canary's verdict — released, held, waived, or auto-skipped — so "was this checked, and what did it say?" is answerable later rather than remembered.

Tests

tests/test_prove_before_buy.py — 12 tests, each refusal provoked on purpose: a check that cannot fail, a check that cannot be executed, a deliverable the filesystem refuses, a bad canary verdict holding the batch, a good canary releasing it, the single-task auto-skip, both waivers, a blank reason, and the record.

Every new test was run against the tree without the fix first, and fails there. A test that has never failed proves nothing.

Full suite: 327 pass (was 315), 1 pre-existing skip.

Two existing tests now waive baseline explicitly rather than being weakened — the budget e2e checks exit 0 on purpose, and the workdir-escape test needs to reach the runtime guard the gate would otherwise pre-empt.

demo keeps its parallel fan-out: the canary is skipped for it with the reason recorded, since serialising the first of three workers would hide the thing the demo exists to show. Its baseline passes cleanly (3 fail, 0 pass, 0 error), verified against the shipped manifest.

Review

Three lens passes. Fixed: the canary verdict was missing from the run record; the unreachable deliverable had no detection at all, not merely no test; unwritable_deliverables overstated what it proves; execute_baseline's docstring said "spawn nothing" when it spawns checks and git helpers (it spawns no workers).

review-accepted: Baseline deliverable probing does not prove worker-sandbox writeability — correct, and the claim is now narrowed everywhere to what the code does. The suggested fix, probing inside the worker sandbox, would require baseline to spawn a worker; spawning none is what makes it cheap enough to run before every dispatch, and is the guarantee the refusal itself depends on. Sandbox scope is already reasoned about statically by lint's worker_unwritable_paths (NateBJones-Projects#129), and what neither can know statically is what the canary is for — one task's spend instead of the manifest's.

review-accepted: Failure-counter docstring contradicts its implementation — failure_counts has zero occurrences in this PR's commits; it belongs to NateBJones-Projects#129 and reached the reviewer through a staging defect in the review kit, fixed separately in bryfa/nexo#259 (work#1034).

Deliberately not built

No baseline caching across runs. The ticket raised skipping the baseline on a re-run of an unchanged manifest. The tree moves under the manifest, so a cached baseline can be green about code that no longer exists — and the baseline spawns no workers, so it is nearly free. Keeping it cheap beats caching a lie.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KX9rgZWnE5zAmJmhaBz35S

[bryfa]

Review pass 1 — and a defect in the review kit itself

Pass 1 returned three findings. Two were real and are fixed in 1a5766e; one is declined. But the pass has to be read with a caveat, because the kit staged the wrong diff.

The staging defect

tools/ringer-kits/codex-review/stage_change.sh computes the diff as:

git diff "$(git merge-base origin/main "$HEAD_SHA")"..."$HEAD_SHA"

It hard-codes origin/main as the base and never consults the PR's actual baseRefName. This PR's base is fix/configurable-check-timeout, and in this checkout origin is the upstream remote — so the merge-base resolved to upstream main and the staged diff became 14 files / 3352 lines, sweeping in all of NateBJones-Projects#74 and NateBJones-Projects#129 alongside my 5 files / 833 lines. The file list matches origin/main...feat/prove-before-buy exactly.

Two of the three "Lens" items were consequently about code in NateBJones-Projects#129, not this PR (worker_unwritable_paths, a ticket-attribution lint rule) — neither symbol exists in this branch's commits.

It fails silently. head.sha was staged correctly (cd9c1260), so review_gate.sh's stale-review protection would have gone green over a review that read 2500 lines of somebody else's code. That is the same failure shape as work#666, one level up: there the sha was right and the diff was stale; here the sha is right and the diff is too large.

This is not specific to a fork — any stacked PR in any repo (base = a feature branch rather than main) gets the same contamination.

The three findings

#	finding	outcome
1	Canary verdict absent from the preflight record	fixed — the verdict now lands at all four decision points (released / held / waived / single-task skip), on the state writer so Preflight stays frozen and pre-dispatch-only
2	The unreachable-deliverable refusal is not tested	fixed, and it was more than a test gap — nothing detected that case at all. Baseline now probes declared absolute expect_files against the nearest existing ancestor. Relative paths are deliberately never probed (they land in the harness-created scratch dir), and that complement is pinned so the gate can't start refusing honest manifests
3	Failure-counter docstring contradicts its implementation	declined — not this change. failure_counts has zero occurrences in either commit here; it belongs to the cost-control work that the mis-staged diff pulled in

review-accepted: Failure-counter docstring contradicts its implementation — the symbol does not appear in this PR's commits; it entered the review through the mis-staged diff described above and belongs to NateBJones-Projects#129.

All three new assertions were run against the pre-fix commit first and fail there. Suite: 327 pass, 1 pre-existing skip.