Skip to content

docs: add out-of-tree task guide and runnable example - #229

Open
jessie1111101 wants to merge 1 commit into
mainfrom
docs/out-of-tree-tasks
Open

docs: add out-of-tree task guide and runnable example#229
jessie1111101 wants to merge 1 commit into
mainfrom
docs/out-of-tree-tasks

Conversation

@jessie1111101

Copy link
Copy Markdown
Collaborator

Follow-up to the agentic-integrity-evals thread, where the ask was "an example or instructions in the README" for defining tasks whose task.yaml and Terraform live in a consumer's own repo.

Docs only — zero source changes. The capability already exists in full; what was missing is a written-down description of it and of the ways it bites.

What's here

  • docs/how-to/out-of-tree-tasks.md — the guide: what can live outside the repo, three ways to handle infra, the external-stack contract, four gotchas, matrix caveats, leaderboard note.
  • examples/out-of-tree-task/ — a directory to copy wholesale into your own repo:
    • noop-task.yamldeployer: noop, runs with no cloud, no credentials, no Docker.
    • task.yaml + tf/deployer: tofu with an absolute stack: path and a self-contained kind stack. kind rather than GCP so anyone with Docker can run it.
  • Index wiring: README.md, docs/README.md, docs/how-to/add-a-task.md, docs/components/infra.md, AGENTS.md, tasks/AGENTS.md.

Corrects one claim

Out-of-tree providers are supported, contrary to what's been said in passing. PROVIDERS carries the entry-point group devops_bench.providers (devops_bench/providers/base.py:27-30), as do METRICS and VERIFIERS. Genuinely in-repo-only: deployers (fixed branch in deployers/factory.py) and agent harnesses (AGENTS registry declares no group). The guide's extension-surface table says so plainly rather than implying everything is pluggable.

The four gotchas

  1. An external stack must name its provider explicitly. _select_provider refuses to deduce it from an absolute path. Empirically reproduced — the ConfigError text in the doc is verbatim from the run.
  2. External stacks don't get per-run isolation. _isolated_work_dir returns the original dir when the stack is outside tf/, so concurrent runs of the same external stack contend on .terraform.lock.hcl. Documented, not fixed — see below.
  3. Relative module sources break outside the repo. source = "../../modules/cluster" doesn't resolve elsewhere; the example shows both the inline and the git-module-source fix.
  4. Undeclared variables are dropped. Provider-injected ones with a log warning; keys from the task's own variables: block raise a ConfigError (they're tracked as custom keys). Empirically reproduced — deleting variable "cluster_name" confirmed the injected name never reaches tofu.

Gotcha 2 is left as-is deliberately. Fixing it is a behavior change with parallel-safety implications and deserves its own PR, not a drive-by in a docs change; it's already tracked as a known hack at docs/appendix/known_issues.md:65. The guide documents current behavior and gives the workaround (per-run copy of the stack dir).

Verification

  • Both task files load from /tmp/oot-demo, outside the repo.
  • get_deployer resolves the absolute stack and passes all five injected vars as -var flags.
  • Gotchas 1 and 4 reproduced (above).
  • tofu validate → valid; tofu fmt -check -recursive examples/ clean.
  • pytest tests/ → 932 passed. Ruff error count identical with and without this change (pre-existing, and this PR adds no Python).
  • MATRIX_TASKS=ALL globs find tasks -name task.yaml, so nothing under examples/ can reach the leaderboard matrix.

Teams evaluating with their own scenarios asked how to keep task definitions
and Terraform in their own repo. The harness already supports this end to end
— an absolute `stack:` path, any CLI `source` path, and entry-point plugins for
providers/verifiers/metrics — but nothing documented it, and `add-a-task.md`
reads as if `tasks/` is the only option.

Add a how-to plus a copyable example. No source changes.

The guide covers the extension surface (including that only deployers and agent
harnesses are in-repo-only; providers, verifiers, and metrics all load from
entry points), the three infra options, the stack output contract, and four
failure modes that each cost a run:

  1. external stacks must name `provider:` explicitly — deduction is in-repo
     only and `_select_provider` raises rather than guess
  2. external stacks don't get the per-run working-dir copy under --parallel,
     so concurrent runs share a `.terraform.lock.hcl`
  3. relative module sources (`../../modules/cluster`) don't resolve off-repo
  4. variables a stack doesn't declare are dropped silently, so a missing
     `variable "cluster_name"` quietly discards the run-unique cluster name

examples/out-of-tree-task/ ships a noop variant that runs with no cloud,
credentials, or Docker, plus a self-contained kind stack demonstrating the
absolute-path flow. `MATRIX_TASKS=ALL` globs `tasks/`, so nothing under
examples/ can reach the leaderboard matrix.
@jessie1111101
jessie1111101 requested review from itssimrank and pradeepvrd and removed request for pradeepvrd July 28, 2026 21:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant