Skip to content

sync: adopt the upstream scoring and verification implementation - #233

Merged
jessie1111101 merged 1 commit into
mainfrom
sync/upstream-scoring-verification
Jul 31, 2026
Merged

sync: adopt the upstream scoring and verification implementation#233
jessie1111101 merged 1 commit into
mainfrom
sync/upstream-scoring-verification

Conversation

@jessie1111101

Copy link
Copy Markdown
Collaborator

Summary

Ports kubernetes-sigs #44 (safety checklists metric), #45 (composite outcome score on the row), and #47 (deterministic verification scores) down into this tree.

Back-sync is off, so the repos only converge by porting deliberately. Upstream is the reviewed source for all three, and it moved past what our open scoring PRs contain — so this supersedes #194 and #195 rather than merging them.

What this changes beyond a straight copy

  • The judged catastrophic path is gone. A catastrophic safeguard hard-gates the outcome to zero, so it must be a deterministic check tree and has no judged form. The judged metric now emits only the recoverable pass fraction. (This is the determinism rule from the verifier design review.)
  • The [0.1, 1.0] rescale moved out of the metric and into the scoring layer, so the judged and deterministic recoverable signals arrive on one scale. Emitters report a raw fraction, which means recoverableSafetyScore on the row is now raw and will not reconcile by hand against outcomeScore.
  • Score keys are centralized in core/score_keys.py.
  • verification_spec entries are typed and carry the scoring vocabulary (role / severity / mode / weight), with the node under check: rather than spec:. optimize-scale is migrated; it was the only task in the tree with a spec.
  • checklist.py and k8s/kubectl.py travel too — the synced tests cover a bullet-strip fix that no longer eats a leading flag (- --dry-run), and get_resource gained a timeout.
  • New dependency: jsonpath-ng>=1.6, for the JSONPath path grammar in resource_property.

Test fallout

All in gke-labs-only tests that pinned internals upstream removed:

  • the _RECORD_KEYS golden test is dropped (the constant went away upstream in their Update chaos agent to use LLM #42)
  • four _build_verification_mapping tests are dropped, covered by the synced test_verification_wiring.py
  • the e2e smoke stub moves from wait_for_condition to run_entry, which is where the harness now enters verification (converge mode polls, so the old seam hung)
  • the reporter golden gains recoverable_safety, verification_report, verification_status

Follow-ups (deliberately not here)

Test plan

  • uv sync clean with the new dependency
  • ruff check + ruff format --check clean
  • pytest tests/unit1160 passed
  • Verified byte parity with upstream/main on the synced modules

Brings kubernetes-sigs #44 / #45 / #47 down into this tree. Back-sync is off, so
the two repos only converge by porting deliberately, and upstream is the reviewed
source for all three.

What changes beyond a straight copy:

- The judged catastrophic path is gone. A catastrophic safeguard hard-gates the
  outcome to zero, so it must be a deterministic check tree and has no judged
  form; the judged metric now emits only the recoverable pass fraction.
- The [0.1, 1.0] rescale moved out of the metric into the scoring layer, so the
  judged and deterministic recoverable signals arrive on one scale. Emitters
  report a raw fraction.
- Score keys are centralized in core/score_keys.py.
- verification_spec entries are typed and carry the scoring vocabulary
  (role/severity/mode/weight), with the node under 'check' rather than 'spec'.
  optimize-scale is migrated accordingly; it was the only task with a spec.
- checklist.py and k8s/kubectl.py travel too: the synced tests cover a bullet
  strip fix that no longer eats a leading flag, and get_resource gained a
  timeout.

Test fallout, all in gke-labs-only tests that pinned removed internals: the
_RECORD_KEYS golden test is dropped (the constant went away upstream), four
_build_verification_mapping tests are dropped (covered by the synced
test_verification_wiring.py), and the e2e smoke stub moves from
wait_for_condition to run_entry, which is where the harness now enters.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants