sync: adopt the upstream scoring and verification implementation - #233
Merged
Conversation
Brings kubernetes-sigs #44 / #45 / #47 down into this tree. Back-sync is off, so the two repos only converge by porting deliberately, and upstream is the reviewed source for all three. What changes beyond a straight copy: - The judged catastrophic path is gone. A catastrophic safeguard hard-gates the outcome to zero, so it must be a deterministic check tree and has no judged form; the judged metric now emits only the recoverable pass fraction. - The [0.1, 1.0] rescale moved out of the metric into the scoring layer, so the judged and deterministic recoverable signals arrive on one scale. Emitters report a raw fraction. - Score keys are centralized in core/score_keys.py. - verification_spec entries are typed and carry the scoring vocabulary (role/severity/mode/weight), with the node under 'check' rather than 'spec'. optimize-scale is migrated accordingly; it was the only task with a spec. - checklist.py and k8s/kubectl.py travel too: the synced tests cover a bullet strip fix that no longer eats a leading flag, and get_resource gained a timeout. Test fallout, all in gke-labs-only tests that pinned removed internals: the _RECORD_KEYS golden test is dropped (the constant went away upstream), four _build_verification_mapping tests are dropped (covered by the synced test_verification_wiring.py), and the e2e smoke stub moves from wait_for_condition to run_entry, which is where the harness now enters.
itssimrank
approved these changes
Jul 31, 2026
This was referenced Jul 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ports kubernetes-sigs #44 (safety checklists metric), #45 (composite outcome score on the row), and #47 (deterministic verification scores) down into this tree.
Back-sync is off, so the repos only converge by porting deliberately. Upstream is the reviewed source for all three, and it moved past what our open scoring PRs contain — so this supersedes #194 and #195 rather than merging them.
What this changes beyond a straight copy
[0.1, 1.0]rescale moved out of the metric and into the scoring layer, so the judged and deterministic recoverable signals arrive on one scale. Emitters report a raw fraction, which meansrecoverableSafetyScoreon the row is now raw and will not reconcile by hand againstoutcomeScore.core/score_keys.py.verification_specentries are typed and carry the scoring vocabulary (role/severity/mode/weight), with the node undercheck:rather thanspec:.optimize-scaleis migrated; it was the only task in the tree with a spec.checklist.pyandk8s/kubectl.pytravel too — the synced tests cover a bullet-strip fix that no longer eats a leading flag (- --dry-run), andget_resourcegained atimeout.jsonpath-ng>=1.6, for the JSONPath path grammar inresource_property.Test fallout
All in gke-labs-only tests that pinned internals upstream removed:
_RECORD_KEYSgolden test is dropped (the constant went away upstream in their Update chaos agent to use LLM #42)_build_verification_mappingtests are dropped, covered by the syncedtest_verification_wiring.pywait_for_conditiontorun_entry, which is where the harness now enters verification (converge mode polls, so the old seam hung)recoverable_safety,verification_report,verification_statusFollow-ups (deliberately not here)
tasks/common/opa-remediationandtf/prebuilt/opa-remediationalso changed upstream; task/stack content is a separate PR.load.mjsvalidatesrecoverableSafetyScoreas[0.1, 1]and would now reject a legitimate0, andPROTOCOL.mddocuments the old range.Test plan
uv syncclean with the new dependencyruff check+ruff format --checkcleanpytest tests/unit— 1160 passedupstream/mainon the synced modules