fix(tasks): make multi-region-failover exercise failover, and cut the prompt to the incident (0.80 → 0.50)

[jessie1111101]

Problem

multi-region-failover provisions two GKE clusters, a global HTTP LB and a cross-region Cloud SQL pair — and then never tests any of it.

The GRADING NOTE said "failover and direct primary recovery are equally acceptable", and nothing prevented an agent from just recreating the node pool the stack deleted. Across six historical runs, gemini-3.1-pro did exactly that on every successful run. The URL map, the standby cluster and the Cloud SQL replica were never touched. The scenario was scenery.

The rubric compounded it. The original criterion 2 bundled replication health, the URL-map flip, node-pool scaling and GitOps reconciliation into one bullet. ChecklistScore is a count of passing checks, so an agent doing one of those four scored identically to one doing all four.

Fixing those two things (first commit) got the agent onto the failover path and the score down to 0.80. That is where this PR originally stopped, and it was still too high — because the prompt had become a runbook.

Change

Prompt and expected_output only. No stack or verification changes.

Prompt: 828 → 373 characters. The long version handed over the Kubernetes namespace, the GitOps repo path, a rewording of the capacity constraint into an explicit instruction ("your recovery must not depend on getting nodes back in that region"), and the exact post-mortem filename. What remains is the service, the project, the symptom, the capacity premise, and "investigate, restore the service, and write a post-mortem."

Which region is impaired, what the user-facing entry point is, where the GitOps repo lives, and what the write path does are now things the agent has to work out.

expected_output: 10 → 11 atomic criteria. The added one is the write path — app-config names a Cloud SQL primary (SQL_PRIMARY) in the impaired region and a read-only replica (SQL_REPLICA) in the standby, so after a failover writes either cross back into the impaired region or have nowhere to go. Either resolution passes: promote the replica, or state that the primary is still reachable because the incident affects compute only and record the cross-region write dependency as a follow-up. Only saying nothing fails. This is the largest thing the 0.80 runs skipped.

Two criteria were also relaxed, because they graded text the shortened prompt no longer contains:

• The post-mortem criterion named incident-report.md. The short prompt does not specify a filename, so any filename now satisfies it.
• The GitOps criterion named ~/app-repo-{{GKE_CLUSTER_NAME}}.git. That placeholder is not interpolated into the prompt — the agent is shown the literal token — so it was being graded on a path it could not have known. The item now matches the repo it finds on disk.

Validation

GKE, openclaw + gemini-3.1-pro-preview, one cluster pair per run, full lifecycle (infra → agent → judge → teardown), every run status: success.

Arm	Prompt	Rubric	Score	Checklist
a3	848 ch	10	0.8000	8/10
a4	824 ch	11 (+write path)	0.6364	7/11
a5	372 ch	11	0.4545	5/11
a6	257 ch (premise cut)	11	0.4545	5/11
a7	257 ch (premise cut)	11, filename relaxed	0.6364	7/11
a8	identical to a7	11, filename relaxed	0.3636	4/11
a9	373 ch — this commit	11	0.5455	6/11
a10	identical to a9	11	0.4545	5/11

This commit is a9/a10: 0.5455 and 0.4545, mean 0.50. Down from 0.80, on a prompt 56% shorter.

The two misses are consistent across both runs and both genuine: no Cloud SQL replication check before the cutover, and no capacity added to the standby before sending it production traffic. Verified against the trajectory — zero gcloud sql calls, zero resize / node-pools calls.

Why not a6/a7/a8

a6 cut the prompt further, to 257 characters, by dropping the capacity premise. That looked like the best result until the transcripts were read: both a7 and a8 failed the criterion "Agent honours the stated capacity constraint" by recreating the East node pool — a constraint their prompt no longer stated. Same class of bug as the incident-report.md filename item. a9 restores the one-sentence premise (still a 56% cut from a3) so that every criterion is grounded in prompt text. The capacity criterion passes in both a9 and a10.

Run-to-run variance is large, and reviewers should know it

a7 and a8 are byte-identical inputs and scored 0.6364 and 0.3636 — a 0.27 spread. a9/a10, also byte-identical, scored 0.5455 and 0.4545 — 0.09.

A single run of this task is not a reliable number. The pair is reported rather than the better of the two, and no claim here rests on a difference smaller than the observed spread. What is stable across all eight arms is the direction: shorter prompt, lower score.

Note for reviewers

This task is judge-only by construction. All three leaf verifiers (pod_healthy, scaling_complete, resource_property) read the world through kubectl. The graded outcome here is a GCE URL map, a Cloud SQL pair and a file on disk. A verification_spec can express roughly 2 of the 11 criteria — and neither of the two the agent actually fails — so converting would score these runs 1.0 again. VerificationCoverage is None and OutcomeScore is ChecklistScore verbatim. Worth tracking separately as a verifier-reach gap.