Skip to content

fix(task): align lustre-csi-deployment expected_output with live deploy - #214

Closed
geojaz wants to merge 1 commit into
gke-labs:mainfrom
onix-net:fix/lustre-csi-task-live-deploy-expected-output
Closed

fix(task): align lustre-csi-deployment expected_output with live deploy#214
geojaz wants to merge 1 commit into
gke-labs:mainfrom
onix-net:fix/lustre-csi-task-live-deploy-expected-output

Conversation

@geojaz

@geojaz geojaz commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Summary

tasks/gcp/lustre-csi-deployment/task.yaml had an inconsistency between its deployer and its expected_output, found during a verifier-coverage audit.

  • The task sets infrastructure.deployer: tofu, so the harness sets generation_only=False (devops_bench/evalharness/default.py:958, where generation_only = deployer == "noop").
  • But its expected_output was written with the noop generation-only template: an Expected Tool Call: generate_manifest line plus a full Expected Manifest Generated: YAML block.

With generation_only=False, the OutcomeValidity GEval judge does not apply the generation-only override. It requires the agent to actually apply resources to a live cluster and confirm the apply (manifest-only is graded a significant failure). The generate_manifest framing contradicted that, so the judge would expect a live apply the criteria never described, mis-scoring the task.

Why this is a live-deploy task

The intent is unambiguously live-deploy, so correcting the expected_output (rather than flipping the deployer) is the minimal fix:

  • It provisions a real GPU cluster (prebuilt/lustre-csi stack, g2-standard-4 node) and references a pre-provisioned Managed Lustre instance.
  • Objective 1 requires querying that live instance for its mount IP and filesystem (gcloud lustre instances describe), which is impossible in a generation-only task.
  • The prompt says "Deploy a live vLLM inference server ... onto the GKE cluster."

Flipping to deployer: noop would be the wrong fix: it would strip the cluster and the Lustre instance and break objective 1.

Change

Rewrite expected_output from the generate_manifest template to an applied-to-cluster success criterion, matching the convention used by the other tofu tasks (tasks/gcp/deploy-config, tasks/gcp/deploy-hello-app).

We are no longer generating a manifest, but the critical requirements checklist still verifies the substantive properties the old manifest block encoded: the Managed Lustre CSI driver, the volumeHandle format, the queried mount IP and filesystem attributes, static PV/PVC binding, the 1-GPU request, the /models mount, and the ClusterIP Service on port 8000. Values the prompt states are enforced; incidental reference-solution values it never states (exact storage-class name, capacity, reclaim policy, target port, replica count) are left out so a valid deployment is not failed for choosing different ones, per the repo's graded-on-outcome convention.

This was the only tofu task using the generate_manifest template.

Testing

  • uv run pytest tests/unit/tasks/test_tasks_schema.py tests/unit/metrics/test_metrics_pipeline.py tests/unit/evalharness/test_reporter.py -q -> 67 passed
  • uv run ruff check . -> no new findings (pre-existing errors are confined to legacy deployers/ and pkg/)
  • YAML re-parses; deployer unchanged (tofu).

The lustre-csi-deployment task uses deployer: tofu, so the harness sets
generation_only=False and the OutcomeValidity judge requires the agent to
apply resources to a live cluster and confirm the apply. But its
expected_output was written with the noop generate_manifest template
(Expected Tool Call: generate_manifest plus a full Expected Manifest
Generated block), which frames it as generation-only. That mismatch made
the judge expect a live apply the criteria never described, mis-scoring the
task.

The task is genuinely a live deploy: it provisions a real GPU cluster and a
pre-provisioned Managed Lustre instance, and objective 1 requires querying
that live instance (gcloud lustre instances describe), which is impossible
in a generation-only task. Rewrite expected_output to an applied-to-cluster
success criterion, matching the convention used by the other tofu tasks
(tasks/gcp/deploy-config and tasks/gcp/deploy-hello-app). This was the only
tofu task using the generate_manifest template.
@geojaz
geojaz force-pushed the fix/lustre-csi-task-live-deploy-expected-output branch from 1e2007d to 349897c Compare July 20, 2026 16:26
@geojaz geojaz closed this Aug 27, 2026
@geojaz
geojaz deleted the fix/lustre-csi-task-live-deploy-expected-output branch August 27, 2026 18:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant