EPIC: reliability-placement study — where does contract compliance live, and how far does it transfer?

[JeremyJC67]

Status

Active roadmap, opened 2026-08-11. Post-v1 follow-on study. This issue is the canonical ordering of
milestones for the next phase.

Question

v1 establishes that observation-contract compliance is measurable and that agents fail it in
characterisable ways. The natural next question is where compliance can be installed, and how far an
installed compliance transfers.

Core claim under test. Reliability is a quantity that can be located. The same contract target can be
installed at different loci in the stack, and the scientific content is where it lives, how much of it
transfers, and at what cost.

Locus	Mechanism	Expected behaviour
prompt	contract stated in context	cheapest; degrades under shift and adversarial pressure
scaffold	verifier / repair loop around the agent	strong in-distribution; brittle to unanticipated constraints
weights	process-supervised post-training (LoRA-scale)	generalises to unseen constraints iff the abstraction was learned, not the form
architecture	typed tool boundary making the violation unrepresentable	strongest guarantee; narrowest coverage

Design: 4 loci x 3 test beds x 3 generalisation distances, over one fixed contract family.
Generalisation distance is the load-bearing factor: (1) same family, unseen instances; (2) same abstract
constraint, disjoint surface form; (3) different abstract constraint.

Milestones

Milestone	Issue	Deliverable	Gate
M0 — reproducibility floor	#2	CI over the suite and the 33 task manifests	current
M1 — perturbation layer	not yet filed	Any task runnable as (task, perturbation, seed), deterministically	current
M2 — form-disjoint families	#4	Two families, same constraints, disjoint surface vocabulary	current
M3 — locus harness	not yet filed	The same contract target installed at all four loci, one runner	M1, #4
M4 — monitorability metric	not yet filed	Counterfactual dependence of outcome on process, per condition	M1
M5 — transfer map	not yet filed	Compliance vs generalisation distance, per locus, with cost	M0-M4

Milestones are filed as issues when their gate opens, not before.

Dependency order

#2 → (M1, #4) → M3 → M4 → M5

Work outside this chain must state which milestone it unblocks, or carry the deferred label.

Pre-registered predictions

Recorded before any run, so that a null result is reportable:

• P1 — at distance (1), all four loci look similar. This is why single-locus results in the literature
  read as successes.
• P2 — at distance (2), the loci separate. This separation is the result; without it there is no
  placement axis.
• P3 — monitorability predicts weight-locus transfer at distance (2) better than in-distribution
  compliance does.

Falsification

If the loci do not separate at distance (2), the placement claim is withdrawn rather than protected with
post-hoc conditions. If monitorability cannot be measured reliably, P3 is dropped and the study reduces to a
three-factor transfer map.

Exit condition

• CI green on main and reproducible from a clean checkout
• perturbation layer determinism test passing
• two form-disjoint contract families, disjointness check enforced
• one contract target installed at all four loci in a single runner
• monitorability measured for every condition, or P3 formally dropped
• transfer map produced with cost per unit of reliability, and P1/P2 adjudicated in writing