docs: add out-of-tree task guide and runnable example - #229
Open
jessie1111101 wants to merge 1 commit into
Open
Conversation
Teams evaluating with their own scenarios asked how to keep task definitions
and Terraform in their own repo. The harness already supports this end to end
— an absolute `stack:` path, any CLI `source` path, and entry-point plugins for
providers/verifiers/metrics — but nothing documented it, and `add-a-task.md`
reads as if `tasks/` is the only option.
Add a how-to plus a copyable example. No source changes.
The guide covers the extension surface (including that only deployers and agent
harnesses are in-repo-only; providers, verifiers, and metrics all load from
entry points), the three infra options, the stack output contract, and four
failure modes that each cost a run:
1. external stacks must name `provider:` explicitly — deduction is in-repo
only and `_select_provider` raises rather than guess
2. external stacks don't get the per-run working-dir copy under --parallel,
so concurrent runs share a `.terraform.lock.hcl`
3. relative module sources (`../../modules/cluster`) don't resolve off-repo
4. variables a stack doesn't declare are dropped silently, so a missing
`variable "cluster_name"` quietly discards the run-unique cluster name
examples/out-of-tree-task/ ships a noop variant that runs with no cloud,
credentials, or Docker, plus a self-contained kind stack demonstrating the
absolute-path flow. `MATRIX_TASKS=ALL` globs `tasks/`, so nothing under
examples/ can reach the leaderboard matrix.
jessie1111101
requested review from
itssimrank and
pradeepvrd
and removed request for
pradeepvrd
July 28, 2026 21:15
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to the agentic-integrity-evals thread, where the ask was "an example or instructions in the README" for defining tasks whose
task.yamland Terraform live in a consumer's own repo.Docs only — zero source changes. The capability already exists in full; what was missing is a written-down description of it and of the ways it bites.
What's here
docs/how-to/out-of-tree-tasks.md— the guide: what can live outside the repo, three ways to handle infra, the external-stack contract, four gotchas, matrix caveats, leaderboard note.examples/out-of-tree-task/— a directory to copy wholesale into your own repo:noop-task.yaml—deployer: noop, runs with no cloud, no credentials, no Docker.task.yaml+tf/—deployer: tofuwith an absolutestack:path and a self-contained kind stack. kind rather than GCP so anyone with Docker can run it.README.md,docs/README.md,docs/how-to/add-a-task.md,docs/components/infra.md,AGENTS.md,tasks/AGENTS.md.Corrects one claim
Out-of-tree providers are supported, contrary to what's been said in passing.
PROVIDERScarries the entry-point groupdevops_bench.providers(devops_bench/providers/base.py:27-30), as doMETRICSandVERIFIERS. Genuinely in-repo-only: deployers (fixed branch indeployers/factory.py) and agent harnesses (AGENTSregistry declares no group). The guide's extension-surface table says so plainly rather than implying everything is pluggable.The four gotchas
_select_providerrefuses to deduce it from an absolute path. Empirically reproduced — theConfigErrortext in the doc is verbatim from the run._isolated_work_dirreturns the original dir when the stack is outsidetf/, so concurrent runs of the same external stack contend on.terraform.lock.hcl. Documented, not fixed — see below.source = "../../modules/cluster"doesn't resolve elsewhere; the example shows both the inline and the git-module-source fix.variables:block raise aConfigError(they're tracked as custom keys). Empirically reproduced — deletingvariable "cluster_name"confirmed the injected name never reaches tofu.Gotcha 2 is left as-is deliberately. Fixing it is a behavior change with parallel-safety implications and deserves its own PR, not a drive-by in a docs change; it's already tracked as a known hack at
docs/appendix/known_issues.md:65. The guide documents current behavior and gives the workaround (per-run copy of the stack dir).Verification
/tmp/oot-demo, outside the repo.get_deployerresolves the absolute stack and passes all five injected vars as-varflags.tofu validate→ valid;tofu fmt -check -recursive examples/clean.pytest tests/→ 932 passed. Ruff error count identical with and without this change (pre-existing, and this PR adds no Python).MATRIX_TASKS=ALLglobsfind tasks -name task.yaml, so nothing underexamples/can reach the leaderboard matrix.