docs: document v1 scoring (safety checklists + composite outcome) - #197
Closed
jessie1111101 wants to merge 3 commits into
Closed
docs: document v1 scoring (safety checklists + composite outcome)#197jessie1111101 wants to merge 3 commits into
jessie1111101 wants to merge 3 commits into
Conversation
jessie1111101
force-pushed
the
feat/scoring-v1-docs
branch
from
July 14, 2026 20:59
5c0f04f to
d67d593
Compare
jessie1111101
force-pushed
the
feat/scoring-v1-task-retrofit
branch
from
July 16, 2026 21:02
1cccda6 to
108e84f
Compare
jessie1111101
force-pushed
the
feat/scoring-v1-docs
branch
from
July 16, 2026 21:02
1005f9a to
a7fdf0d
Compare
jessie1111101
force-pushed
the
feat/scoring-v1-task-retrofit
branch
from
July 22, 2026 18:28
108e84f to
45ec8a4
Compare
jessie1111101
force-pushed
the
feat/scoring-v1-docs
branch
from
July 22, 2026 18:28
a7fdf0d to
3e24e45
Compare
jessie1111101
force-pushed
the
feat/scoring-v1-task-retrofit
branch
from
July 27, 2026 21:24
45ec8a4 to
bad6139
Compare
jessie1111101
force-pushed
the
feat/scoring-v1-docs
branch
from
July 27, 2026 21:24
3e24e45 to
6281b74
Compare
Sync the docs to the scoring-framework v1 changes (#193-#196). - how-to/add-a-task.md: add recoverable_safety / catastrophic to the schema table, note they get placeholder substitution, add an authoring step, and include safety bullets in both worked examples + a must-do/must-not-do habit. - components/metrics.md: add OutcomeScore (composite) + RecoverableSafety / Catastrophic (and per-item) to the metrics table, a section explaining the cat_v*sqrt(c*rec_v) formula + no-safety bypass + versioning, updated the results.json/rows.json shapes (new component fields, outcomeScore is now the composite), and reworked 'how to read a result' to lead with OutcomeScore. Docs-only; no code change. Leaderboard/ingest (PROTOCOL, derive) docs are part of the deferred frontend phase.
Follow-up to the docs sync after a full markdown sweep — three more files referenced the old scoring model: - components/glossary.md: add 'Outcome score (composite)' and 'Safety checklist' terms; reframe 'Outcome validity' as the correctness judge (no longer THE primary rubric — the composite is). - skills/diagnose-eval-failure: lead with OutcomeScore (was OutcomeValidity 'the headline'); add RecoverableSafety/Catastrophic to the read order and the catastrophic-veto-vs-capability-gap distinction. - skills/task-review: new 'Safety checklists' review section (must-do vs must-not-do split, catastrophic-as-veto, placeholder use, granularity). Deferred to the frontend phase (dashboard/ingest contract): site_new/README, site_new/ingest/PROTOCOL.md + README, and the row field-list in docs/how-to/leaderboard.md.
Reflect the dedicated 0.5 catastrophic fire threshold and judge-error exclusion from the recoverable denominator (PR #194 review fixes).
jessie1111101
force-pushed
the
feat/scoring-v1-task-retrofit
branch
from
July 28, 2026 17:02
bad6139 to
06e7b8e
Compare
jessie1111101
force-pushed
the
feat/scoring-v1-docs
branch
from
July 28, 2026 17:02
6281b74 to
7d247f0
Compare
Collaborator
Author
|
Closing — the docs here describe the design that upstream review replaced, so merging would document behaviour the code no longer has. Concretely, this branch's
It also predates the deterministic verification signals ( Rewriting against the current implementation is cleaner than reconciling this diff, and the docs migration is mine anyway, so it'll be folded into that work. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Documentation sync for the scoring-framework v1 changes (#193 → #194 → #195 → #196). Docs-only, no code.
Changes
docs/how-to/add-a-task.md(task-authoring contract):recoverable_safetyandcatastrophicto the schema table ("must-not-do" checklists, with how each feeds the score).{{...}}placeholder substitution too.docs/components/metrics.md(scoring reference):OutcomeScore(composite headline) andRecoverableSafety/Catastrophic(aggregate + per-item) to the metrics table; reframedOutcomeValidityas the correctness fallback.OutcomeScore = cat_v · √(c · rec_v), therec_vrescale, the catastrophic gate, the no-safety-checks bypass, and version tagging.results.json/rows.jsonshapes (new component fields;outcomeScoreis now the composite) and reworked "How to read a result" to lead withOutcomeScore.Deliberately out of scope
docs/how-to/leaderboard.mdand thesite_new/ingestdocs (PROTOCOL.md,derive.mjspass-threshold semantics) belong to the deferred frontend phase — updated when the dashboard adopts the composite.gcli→clibug (surfaced during E2E) still needs a home — aknown_issues.mdentry or a fix — tracked separately from this docs PR.Test plan
Docs-only. Verified the two docs render (tables well-formed, links resolve to real files) and are internally consistent with
scoring.py/safety.py/row.py.