Skip to content

Build paper tables from frozen audit JSON, not from demo stdout #13

Description

@esaran1

Why

asla demo is a narrative smoke test. asla report and scripts/run_paper_experiments.py exist, but paper numbers for the real grid should come from a frozen results/audit.json plus a deterministic table generator. Synthetic experiment scripts are not a substitute for the 45-job audit.

What to do

Add something like scripts/make_tables.py that reads audit JSON and writes:

  • CSV and LaTeX for projection vs single-scale vs gate (if included); ensemble/race only if those rankers were actually evaluated on the same budget contract
  • estimand name, n_boot, seed, input SHA-256, fit form, budget roles
  • under-seeded cell list
  • crossover table with fit is blind preserved as missing, not as a dash filled in later

Also generate a one-page methods box: resampling unit, noise-band definition, gate rule, and what claims are not licensed.

Acceptance

  • Re-running the script on the same JSON is bit-identical
  • Tables cannot be produced from --fast without labeling them as smoke tests
  • None crossover budgets stay blank/None in LaTeX
  • README documents the exact command used for the paper tables

Do not fabricate

  • Copy numbers by hand from demo terminal output
  • Leave Infinity or NaN in paper artifacts
  • Treat scripts/run_paper_experiments.py --fast synthetic output as the real-grid result

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentationpaperPaper tables, figures, and write-up

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions