Skip to content

docs: document v1 scoring (safety checklists + composite outcome) - #197

Closed
jessie1111101 wants to merge 3 commits into
feat/scoring-v1-task-retrofitfrom
feat/scoring-v1-docs
Closed

docs: document v1 scoring (safety checklists + composite outcome)#197
jessie1111101 wants to merge 3 commits into
feat/scoring-v1-task-retrofitfrom
feat/scoring-v1-docs

Conversation

@jessie1111101

Copy link
Copy Markdown
Collaborator

Summary

Documentation sync for the scoring-framework v1 changes (#193#194#195#196). Docs-only, no code.

Stacked on #196. Merge order: #193#194#195#196 → this.

Changes

docs/how-to/add-a-task.md (task-authoring contract):

  • Added recoverable_safety and catastrophic to the schema table ("must-not-do" checklists, with how each feeds the score).
  • Noted these fields get {{...}} placeholder substitution too.
  • New authoring step for safety checklists; safety bullets added to both worked examples (noop + tofu).
  • New "split must-do from must-not-do" habit under Key considerations.

docs/components/metrics.md (scoring reference):

  • Added OutcomeScore (composite headline) and RecoverableSafety / Catastrophic (aggregate + per-item) to the metrics table; reframed OutcomeValidity as the correctness fallback.
  • New section explaining OutcomeScore = cat_v · √(c · rec_v), the rec_v rescale, the catastrophic gate, the no-safety-checks bypass, and version tagging.
  • Updated the results.json / rows.json shapes (new component fields; outcomeScore is now the composite) and reworked "How to read a result" to lead with OutcomeScore.

Deliberately out of scope

  • docs/how-to/leaderboard.md and the site_new/ingest docs (PROTOCOL.md, derive.mjs pass-threshold semantics) belong to the deferred frontend phase — updated when the dashboard adopts the composite.
  • The pre-existing matrix-wrapper gcli→cli bug (surfaced during E2E) still needs a home — a known_issues.md entry or a fix — tracked separately from this docs PR.

Test plan

Docs-only. Verified the two docs render (tables well-formed, links resolve to real files) and are internally consistent with scoring.py / safety.py / row.py.

@jessie1111101
jessie1111101 force-pushed the feat/scoring-v1-docs branch from 5c0f04f to d67d593 Compare July 14, 2026 20:59
@jessie1111101
jessie1111101 force-pushed the feat/scoring-v1-task-retrofit branch from 1cccda6 to 108e84f Compare July 16, 2026 21:02
@jessie1111101
jessie1111101 force-pushed the feat/scoring-v1-docs branch from 1005f9a to a7fdf0d Compare July 16, 2026 21:02
@jessie1111101
jessie1111101 force-pushed the feat/scoring-v1-task-retrofit branch from 108e84f to 45ec8a4 Compare July 22, 2026 18:28
@jessie1111101
jessie1111101 force-pushed the feat/scoring-v1-docs branch from a7fdf0d to 3e24e45 Compare July 22, 2026 18:28
@jessie1111101
jessie1111101 force-pushed the feat/scoring-v1-task-retrofit branch from 45ec8a4 to bad6139 Compare July 27, 2026 21:24
@jessie1111101
jessie1111101 force-pushed the feat/scoring-v1-docs branch from 3e24e45 to 6281b74 Compare July 27, 2026 21:24
Sync the docs to the scoring-framework v1 changes (#193-#196).

- how-to/add-a-task.md: add recoverable_safety / catastrophic to the schema
  table, note they get placeholder substitution, add an authoring step, and
  include safety bullets in both worked examples + a must-do/must-not-do habit.
- components/metrics.md: add OutcomeScore (composite) + RecoverableSafety /
  Catastrophic (and per-item) to the metrics table, a section explaining the
  cat_v*sqrt(c*rec_v) formula + no-safety bypass + versioning, updated the
  results.json/rows.json shapes (new component fields, outcomeScore is now the
  composite), and reworked 'how to read a result' to lead with OutcomeScore.

Docs-only; no code change. Leaderboard/ingest (PROTOCOL, derive) docs are
part of the deferred frontend phase.
Follow-up to the docs sync after a full markdown sweep — three more files
referenced the old scoring model:

- components/glossary.md: add 'Outcome score (composite)' and 'Safety
  checklist' terms; reframe 'Outcome validity' as the correctness judge
  (no longer THE primary rubric — the composite is).
- skills/diagnose-eval-failure: lead with OutcomeScore (was OutcomeValidity
  'the headline'); add RecoverableSafety/Catastrophic to the read order and
  the catastrophic-veto-vs-capability-gap distinction.
- skills/task-review: new 'Safety checklists' review section (must-do vs
  must-not-do split, catastrophic-as-veto, placeholder use, granularity).

Deferred to the frontend phase (dashboard/ingest contract): site_new/README,
site_new/ingest/PROTOCOL.md + README, and the row field-list in
docs/how-to/leaderboard.md.
Reflect the dedicated 0.5 catastrophic fire threshold and judge-error exclusion
from the recoverable denominator (PR #194 review fixes).
@jessie1111101
jessie1111101 force-pushed the feat/scoring-v1-task-retrofit branch from bad6139 to 06e7b8e Compare July 28, 2026 17:02
@jessie1111101
jessie1111101 force-pushed the feat/scoring-v1-docs branch from 6281b74 to 7d247f0 Compare July 28, 2026 17:02
@jessie1111101

Copy link
Copy Markdown
Collaborator Author

Closing — the docs here describe the design that upstream review replaced, so merging would document behaviour the code no longer has. Concretely, this branch's metrics.md still says:

  • RecoverableSafety is the score key, rescaled to [0.1, 1.0] by the metric. It's now JudgedRecoverable, carrying the raw fraction, with the rescale applied by the scoring layer.
  • Catastrophic: <item> is an LLM judge that "fires < 0.5". The judged catastrophic path no longer exists; catastrophic safeguards are deterministic only.
  • The example results.json block emits keys that aren't produced anymore.

It also predates the deterministic verification signals (VerificationCorrectness / VerificationRecoverable / VerificationCatastrophic / VerificationCoverage) that now feed the composite.

Rewriting against the current implementation is cleaner than reconciling this diff, and the docs migration is mine anyway, so it'll be folded into that work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant