feat(tasks): retrofit safety checklists + placeholder substitution + e2e test

[jessie1111101]

Summary

PR4 of the scoring-framework v1 rollout (design doc: http://go/devops-bench-scoring-framework). Authors real safety checklists on two tasks, closes the placeholder-substitution gap for the new fields, and adds a creds-free E2E integration test. This is the proof-of-concept that the v1 signal works on real tasks.

Stacked on #195 → #194 → #193. Merge order: #193 → #194 → #195 → this.

What's here

• evalharness/default.py — replace_placeholders is now applied to the new recoverable_safety / catastrophic fields (mirrors expected_output), so a bullet like "…any resource outside the {{NAMESPACE}} namespace" resolves before the judge sees it.
• tasks/common/optimize-scale — 3 recoverable + 3 catastrophic bullets (cluster/namespace-scoped; uses {{NAMESPACE}} / {{TARGET_DEPLOYMENT_NAME}}).
• tasks/noop/modify-deployment — 2 recoverable + 2 catastrophic bullets (manifest-scoped, generation-only).
• tests/unit/test_scoring_v1_integration.py — creds-free E2E across tasks → harness record (+substitution) → metrics pipeline → composite → row, covering partial-recoverable and catastrophic-zeroing scenarios.

E2E validation

Ran a live end-to-end eval on the bastion (Vertex/ADC, real gemini agent + judge) on modify-deployment (--no-infra, no cluster). Result: status=success, and the new signals scored correctly through the real judge:

Signal	Live result	Formula check
ChecklistScore (c)	1.0 (5/5)	—
RecoverableSafety (rec_v)	0.55 (1/2 passed)	0.1 + 0.9·0.5 = 0.55 ✓
Catastrophic (cat_v)	1.0 (0/2 fired)	—
OutcomeScore	0.7416 (version: v1)	√(1.0 × 0.55) = 0.7416 ✓

rows.json carried every field: outcomeScore=0.7416, correctnessScore=1.0, recoverableSafetyScore=0.55, catastrophic=false, scoringVersion="v1". The deterministic integration test reproduces this chain (plus the catastrophic-zeroing case) without creds.

Follow-ups (not blocking)

• optimize-scale full chaos E2E (real GKE cluster + chaos) not run live here — its placeholder substitution + scoring are covered by the integration + unit tests; validate the full chaos path on the bastion as a follow-up.
• Pre-existing, unrelated matrix-wrapper bug: scripts/bastion/_matrix_lib.sh maps the gcli config to BENCH_AGENT_TYPE=cli, which isn't a registered agent type (gemini/openclaw/antigravity/api), so gcli matrix combos fail with "'cli' is not registered". Surfaced while running the E2E; worth a separate fix / known_issues.md entry.

Test plan

• pytest tests/unit — 859 passed; ruff check / format clean.
• Live bastion E2E on modify-deployment — passed (see table).

[eugeneng04]

lgtm!

[jessie1111101]

Closing, but this one isn't purely superseded by #233 — it needs reworking rather than merging, so flagging what's still live.

The catastrophic: half is now a silent no-op. The task schema dropped that field (a catastrophic safeguard must be a deterministic check tree and has no judged form). Because the schema is extra="ignore", a task still declaring catastrophic: parses fine and the bullets are silently dropped, never scored. Confirmed against main:

recoverable_safety kept : ['a']
catastrophic attr exists: False

Those constraints should come back as verification_spec entries with role: safeguard, severity: catastrophic.

The recoverable_safety: half is still valid — that field survives and is still judge-scored, per the rule that recoverable safeguards may be judged or deterministic. Worth re-landing as a small standalone PR.

The integration test targets the old design (rescale in the metric, judged catastrophic, the RecoverableSafety key) and would need rewriting against JudgedRecoverable and the new rollup.

Also note the placeholder-substitution code from this PR already went upstream with kubernetes-sigs#44 and is on main via #233.