napkin-math: advisory source-preservation audit (Fork B, proposal 141 PR 1)

[neoneye]

Summary

Starts proposal 141 with the smallest useful slice: a deterministic advisory audit that diffs a prior parameters.json against a current parameters.json and classifies every prior signal as one of five outcomes. No strict mode, no CI gating, no schema changes to extract prompts, no LLM rationale parsing — those land in later proposal 141 PRs after this audit's findings (false-positive rate, useful catches) are measured on the corpus.

What counts as a "prior signal"

Both entry id values AND entry output_name values across the five sections (key_values, missing_values_to_estimate, derived_questions, recommended_first_calculations, unmodelled_gates). Output_names are first-class because downstream consumers (calculations.py, monte-carlo, summarize-assessment) bind by output_name, not by entry id — a calc whose entry id survives but whose output_name changes is a genuine signal regression.

Classification

Each prior signal name is classified by what evidence the current artifact provides:

Status	Meaning
preserved_by_id	same name appears as a current entry id
preserved_by_output_name	name appears as a current output_name (downstream binders use output_name)
preserved_as_formula_dependency	name is on a current formula_hint RHS or in a current depends_on list
likely_renamed	snake_case token Jaccard overlap ≥ 0.4 with one or more candidates from current ids ∪ current output_names (advisory; top 3 candidates by overlap)
absent_unexplained	no preservation evidence at all

Threshold calibration: actual_X → X_target overlaps at 0.6, X_dkk → X_floor_dkk at 0.75, X_capex → X_capex_floor at 0.83. Unrelated ids stay near 0. 0.4 is permissive enough to surface renames without flooding the report.

Detail entries carry prior_kind ("id" or "output_name") so reviewers can distinguish entry-id drift from output-name drift. The text renderer tags each line with [section/kind].

Out of scope (later proposal 141 PRs)

• Fork A: source-digest regex scan against the current artifact.
• dropped_signals schema: optional field in extract prompts.
• LLM rationale parsing of dropped_signals entries.
• Strict mode / CI gating policy.

Empirical findings on v49 → v51 (6 probes)

Plan	Prior signals	by_id	by_output_name	renamed	absent
euro_adoption	18	17	0	1	0
yellowstone	18	12	0	5	1
crate	23	11	2	7	3
mars_gtld	23	14	1	6	2
datacenter	25	0	0	11	14
paperclip	18	3	0	12	3

Useful catches the audit surfaces mechanically:

• Paperclip latency-tripwire trio all absent from v51: actual_api_p99_latency_ms, api_latency_margin_ms, api_latency_p99_threshold_ms — exactly the silent-regression failure mode the plan doc has been flagging as needing mechanical detection.
• Yellowstone public_compliance_threshold absent — documented v49→v51 regression in the plan's "Known limitations" section.
• Crate q_logistics_budget_margin, target_recovery_rate, annual_crate_loss_volume all absent.
• Datacenter wholesale restructure (0 ids preserved verbatim, 11 renamed, 14 absent) — useful diagnostic; reviewer can scan rename candidates.

False-positive surface (called out honestly):

• Some likely_renamed candidates at jaccard 0.4–0.5 are borderline. Example: paperclip's manual_intervention_margin_hours → actual_manual_intervention_hours_per_week (j=0.43) might be a different concept (margin variable vs realised variable), not a true rename.
• The token-only heuristic treats role changes (margin ↔ realised) as renames.
• No semantic check (label overlap, source_text overlap) — just snake_case token Jaccard. Adding semantic checks is a candidate refinement for PR 3 once we see how reviewers actually use the audit.

Test plan

• pytest experiments/napkin_math/tests/test_audit_source_preservation.py (17 synthetic unit tests pass — 15 + 2 for output_name drift and output_name-as-rename-candidate)
• Audit run end-to-end on all 6 v49→v51 probes — output is sensible and surfaces the known absences
• No hand-patching of parameters.json (the audit reads, never writes)
• Output_name treated as a first-class audited signal (review feedback)
• CI green on this branch

🤖 Generated with Claude Code

[neoneye]

Addressed (`266b6af2`). Output_name is now a first-class audited signal:

1. build_signal_index returns a combined signals dict containing both ids AND output_names. The audit iterates this combined set instead of just prior ids.
2. find_rename_candidates searches the union of current ids and current output_names so Mars-style restructures (where a prior id is moved to a new calc's output_name) surface as likely_renamed.
3. Detail entries gain prior_kind ("id" or "output_name"); text report tags each line with `[section/kind]`.

The exact test you requested is in: test_output_name_drift_on_preserved_id_is_reported. Prior derived_question{id=q_margin, output_name=old_margin} against current derived_question{id=q_margin, output_name=new_margin} reports q_margin preserved_by_id AND old_margin absent_unexplained, with prior_kind=\"output_name\" on the second detail entry.

Also added test_rename_candidates_drawn_from_output_names_too — a Mars-style rename where the prior id maps to a current output_name (not a current id) now surfaces as likely_renamed with the output_name as a candidate.

Corpus probe re-run (counts in updated PR body): prior_total grew for plans whose prior output_names were distinct from their ids — crate 21→23, mars_gtld 22→23, datacenter 23→25. Euro_adoption, yellowstone, and paperclip totals are unchanged because their prior calcs all had id == output_name. All previously-surfaced absences (paperclip latency trio, yellowstone public_compliance_threshold, crate trio, datacenter restructure) still surface; no findings regressed.

17 unit tests pass total (15 prior + 2 new). PR body updated.