Skip to content

DOCKER_ISOLATION.md

Latest commit

 

History

History
342 lines (269 loc) · 20.2 KB

DOCKER_ISOLATION.md

File metadata and controls

342 lines (269 loc) · 20.2 KB
Raw
description Run each Coder Eval task in its own fresh Docker container — strong host isolation, a pinned reproducible agent runtime, and custom images for task-specific dependencies.

Docker Isolation

Run each evaluation task inside its own fresh container. Strong host isolation and a pinned, reproducible agent runtime.

When to use

Set sandbox.driver: docker on a task (or pass --driver docker on the CLI — a thin alias for -D sandbox.driver=docker) when you want:

  • Isolation from the host filesystem/network — agent-generated code can't reach files outside the sandbox.
  • A pinned toolchain — the image bakes in Python 3.13, Node 22 LTS, @anthropic-ai/claude-code, uv, and the matching coder_eval version, so results don't drift with host upgrades.

Aggregation (P/R/F1, suite thresholds, reports) always stays on the host. Each container is a sealed "run one task → emit one task.json" worker.

One-time setup

make docker-image        # core + both built-in agents (default; no credentials)
# opt in to the UiPath extra (resolves from public PyPI; no credentials):
make docker-image-full

Both build coder-eval-agent:<pkg-version> and tag it :latest.

  • make docker-image installs the core package plus both built-in agents — claude-code (baked above) and Codex (--extra codex, public PyPI). It needs no credentials and covers the common case: claude-code or Codex tasks scored with run_command / file_contains (incl. converted skillsbench tasks). llm_judge / agent_judge work here too (they route through the run's Anthropic/Bedrock backend).
  • make docker-image-full additionally installs the uipath extra. The uipath SDK resolves from public PyPI (per uv.lock), so the build needs no credentials. Use this only for tasks that shell out to the in-host uipath CLI. (Codex is already in the default image — no extra needed.)

Codex sandbox under Docker. Codex's Landlock-backed read-only / workspace-write sandboxes can't initialize inside the eval container — their writes/execs fail silently and the agent produces no artifacts (a score=0 FAILURE with no loud error). The docker runner sets CODER_EVAL_IN_CONTAINER=1, and the Codex agent honors it by falling back to full-access: the container itself is the trust boundary. Host runs (tempdir) are unaffected — Landlock works there and the marker is unset. So Codex tasks run under --driver docker with their natural acceptEdits permission mode; no need to set bypassPermissions by hand.

Running a task in Docker

# Single task
coder-eval run path/to/task.yaml --driver docker

# All tasks
coder-eval run --driver docker

# Or in the task YAML
sandbox:
  driver: docker
  docker:
    network: bridge         # or "none" for sealed runs
    image: my-custom:tag    # override the default image

Using a pre-built custom image

When your tasks need extra tools or dependencies, extend the framework image once and point tasks at the result — no per-task build. The custom image must extend coder-eval-agent:<version> so it inherits the coder-eval runtime, the entrypoint script, and the org.coder-eval.version label the host's preflight check reads. (For a task whose Dockerfile can't be rebased onto Debian, use the runtime kit instead. To have coder-eval build the image per task rather than pre-building it, see Building the image from a task Dockerfile.)

FROM coder-eval-agent:<version>          # match the version your host runs
RUN apt-get update && apt-get install -y --no-install-recommends custom-tool \
    && rm -rf /var/lib/apt/lists/*
docker build -t my-team/image:latest .

Select it either in the task YAML:

sandbox:
  driver: docker
  docker:
    image: my-team/image:latest

…or from the CLI, which overrides whatever the YAML says:

coder-eval run task.yaml -D sandbox.docker.image=my-team/image:latest

The default when you set nothing is coder-eval-agent:<installed package version>.

A worked example ships in-tree: tasks/byod_smoke_test.yaml runs against templates/byod_smoke_test/Dockerfile, which extends the framework image and drops a marker file that the task's success criterion then asserts — proving the custom image was actually used. (The byod_* names here mean "Bring Your Own Docker"; they are unrelated to the Bring Your Own Dataset guide, which is about fanning one task out over data rows.)

make docker-image                                                   # base image first
docker build -t byod-custom-image:0.1.0 templates/byod_smoke_test/  # then the derived one
coder-eval run tasks/byod_smoke_test.yaml

Troubleshooting custom images

Symptom Cause and fix
docker: Error response from daemon: pull access denied The image isn't built locally and isn't pullable. Check docker images, then rebuild it. Docker treats an unknown local tag as a remote reference, which is why the error mentions a pull.
Image <your-image> coder_eval <a> != host <b> The custom image carries an org.coder-eval.version label inherited from a stale framework base. Rebuild the base with make docker-image, then rebuild your derived image with docker build --no-cache.
Image <your-image> has no org.coder-eval.version label The image doesn't descend from coder-eval-agent (or predates the label). Rebase it on the framework image, or use the runtime kit.

Building the image from a task Dockerfile

Instead of pointing at a pre-built image, a task can ship its own Dockerfile and have coder-eval build it before the run:

sandbox:
  driver: docker
  docker:
    dockerfile_path: ./environment/Dockerfile   # relative to the task YAML

⚠️ Contract: a task Dockerfile MUST either start with FROM coder-eval-agent:<version> or use the runtime kit (see below). The container runs the coder-eval orchestrator (coder-eval _run-task-internal) via the framework image's ENTRYPOINT. A task Dockerfile extends that image and adds only task-specific layers — extra apt packages, COPY-ed inputs, etc.

Build the framework base first — make docker-image (tags both coder-eval-agent:<version> and coder-eval-agent:latest).

Tasks that bring their own base image: the runtime kit (coder-eval-runtime)

The FROM coder-eval-agent contract above means a task is rebased onto the Debian framework image. That breaks tasks whose Dockerfile was written for a different base image (e.g. a Fedora recipe using dnf, which doesn't exist on Debian). To keep the task's own base image and build successfully, coder-eval's runtime need to be copied into the task's image. Use make coder-eval-runtime first to make the runtime available for copying.

FROM fedora:41                 # the task's own base, kept verbatim
RUN dnf -y install ...         # the task's native recipe, runs on its own OS
# --- copy coder-eval runtime into a task image ---
COPY --from=coder-eval-runtime:latest /opt/coder-eval /opt/coder-eval
COPY --from=coder-eval-runtime:latest /usr/local/bin/coder_eval_entrypoint.sh /usr/local/bin/coder_eval_entrypoint.sh
LABEL org.coder-eval.version="<ver>"

make coder-eval-runtime builds the kit (docker/Dockerfile.runtime): a standalone CPython + Node + the coder-eval CLI + Claude Code, all under /opt/coder-eval, plus the entrypoint at the same /usr/local/bin/... path the host pins. The kit is glibc-only — it runs on debian/ubuntu/fedora/rhel/… but not musl/Alpine.

Both base images are independent and persistent — build each once and run any mix of rebase and inject tasks without rebuilding. To build both in one shot (no credentials needed), use make docker-images (= make docker-image + make coder-eval-runtime); reach for the individual targets when you only need one.

The kit installs the no-credential set only (core + codex), like make docker-image — it never installs the [uipath] extra, so there is no make docker-images-full. An inject-mode task that needs the LLMGW/uipath judge isn't supported by the kit as built.

# environment/Dockerfile
FROM coder-eval-agent:latest          # inherit runtime + entrypoint
RUN apt-get update && apt-get install -y --no-install-recommends poppler-utils
RUN pip install --no-cache-dir PyMuPDF==1.24.10
COPY input/ /root/input/

Behavior:

  • Path resolution — dockerfile_path is resolved relative to the task YAML's directory at load time (with $VAR / ${VAR} expansion). A missing file fails fast at load, not mid-run.
  • Entrypoint check — after building, coder-eval inspects the image's ENTRYPOINT and aborts with a FROM coder-eval-agent hint if the runtime wasn't inherited.
  • Overrides image — when dockerfile_path is set, it takes precedence over any image value.
  • Build context — the build context is the Dockerfile's parent directory, so relative COPY ./input/... instructions resolve naturally. In the layout above, environment/ is the context.
  • Caching — the image is tagged deterministically as coder-eval-task-<task_id>:built, so repeat runs of the same task reuse Docker's layer cache. Edit the Dockerfile and the next run rebuilds the changed layers only.
  • Version-label check skipped — the org.coder-eval.version preflight only applies to the framework image; task-built images don't carry it and won't warn.

A build failure aborts the task with a DockerBuildError (a DockerRunError subclass) carrying docker build's output. Because the build runs before the run dir, docker.log, or task.json exist, the runner explicitly records the failure so it is never a silent empty result dir: it creates the run dir, writes the full build log to docker.log, and writes a synthetic task.json with final_status: BUILD_FAILED (an error-category status) before re-raising. So a failed build shows up per-task on the dashboard with its build log, exactly where you'd look for container output.

Customizing the build (docker.build)

The docker build invocation is configurable via sandbox.docker.build:

sandbox:
  driver: docker
  docker:
    dockerfile_path: ./environment/Dockerfile
    build:
      args:                          # -> --build-arg KEY=VALUE
        PKG_VERSION: "1.2.3"
        TOKEN: "${HOST_TOKEN}"       # values are $VAR / ${VAR} expanded from the host env
      secrets:                       # -> --secret <spec> (requires BuildKit)
        - id=mytoken,env=MY_TOKEN    # forward a host env var as a build secret
        - id=npmrc,src=~/.npmrc      # or a file
      extra_args: ["--target", "runtime"]   # escape hatch for any other docker build flag
      buildkit: true                 # optional: force DOCKER_BUILDKIT (see below)
  • args → --build-arg KEY=VALUE. Values are environment-expanded against the host. Prefer secrets for credentials — build-args are recorded in the image history.
  • secrets → --secret <spec>. Use id=NAME,env=VAR to forward a host env var or id=NAME,src=PATH for a file; reference it in the Dockerfile via RUN --mount=type=secret,id=NAME .... Secrets are exposed only to the mounting RUN step and never baked into layers. Secrets require BuildKit.
  • extra_args → raw flags inserted before the build context (e.g. --target, --network, --platform). Escape hatch for options without a dedicated field.
  • buildkit → controls the DOCKER_BUILDKIT env var. Omitted (default), coder-eval inherits the invoker's environment — set DOCKER_BUILDKIT=1 before running coder-eval to enable it globally. Set buildkit: true / false to force it per task. If secrets are configured but BuildKit isn't enabled, coder-eval logs a warning (the build would otherwise fail).

The build context is always appended last, so extra_args can't displace it.

Authentication

macOS users — read this first. Claude Code's OAuth tokens live in the macOS Keychain. The container has no path to the Keychain, so the bundled CLI inside will return Not logged in · Please run /login and every task will fail at iteration 1. Before running --driver docker, set one of these on the host:

  • ANTHROPIC_API_KEY=... (direct Anthropic), or
  • CLAUDE_CODE_USE_BEDROCK=1 + AWS_BEARER_TOKEN_BEDROCK=... + AWS_REGION=... (Bedrock).

Linux hosts where Claude Code stores creds under ~/.claude already work because that directory is bind-mounted into the container.

Credentials are forwarded via --env VAR (name-only, never embedded in argv) for these vars when set on the host: ANTHROPIC_API_KEY, API_BACKEND, UIPATH_*, AWS_BEARER_TOKEN_BEDROCK, AWS_REGION, CLAUDE_CODE_USE_BEDROCK, ANTHROPIC_MODEL.

To add one or two custom vars to the defaults (recommended), use env_passthrough_extra:

sandbox:
  driver: docker
  docker:
    env_passthrough_extra: ["MY_CUSTOM_TOKEN", "DEBUG_FLAG"]  # Keeps all defaults + these

To completely replace the list, use env_passthrough:

sandbox:
  driver: docker
  docker:
    env_passthrough: ["MY_CUSTOM_TOKEN", "ANTHROPIC_API_KEY"]

HOME is forwarded by default

The default env_passthrough includes HOME so the in-container ~/.claude lookup resolves at the same path as on the host (the mount lands at $HOME/.claude symmetrically). Practical contract:

  • Path.home() inside the container returns the host's HOME value (e.g. /Users/you on macOS). The directory exists in the container because Docker auto-creates it as the mount parent for ~/.claude.
  • ~/.claude is not the host's real dir — the runner makes a throwaway lean copy in a tmp dir per task and mounts that copy read-write at $HOME/.claude. The copy keeps the small set the container needs (auth via .credentials.json, settings.json, plugins/) and drops heavy or transient per-session state — security/ (often hundreds of MB), projects/, cache/, file-history/, backups/, downloads/, sessions/, telemetry/, shell-snapshots/, todos/, session-env/, plus the volatile churn dirs the live CLI rewrites. The skip set is a denylist; the authoritative list is CLAUDE_COPY_IGNORE in src/coder_eval/isolation/docker_runner.py (a test asserts this doc and that constant agree, so the list never silently drifts). The container may write anywhere under ~/.claude; those writes hit the copy and are discarded when the task ends — the host's real ~/.claude is never modified. Note the copy includes the OAuth token (.credentials.json) and is mounted read-write, so the in-container agent can read and tamper with the token copy — contained, since the copy is discarded at task end and the host's real dir is untouched. Opt out entirely with CODER_EVAL_NO_CLAUDE_MOUNT=1.
  • Writes under $HOME outside the ~/.claude mount land in the container's ephemeral rootfs overlay. Don't expect them to persist or to be visible to the host.
  • If a tool detects platform from HOME (e.g. "starts with /Users/ → macOS"), it will draw the wrong conclusion. Vanishingly rare in practice.

Remove HOME from env_passthrough if you don't want this behavior — the container's image-default HOME=/root will win, but then the host's OAuth dir is no longer reachable.

Run directory safety (--run-dir)

The host's run dir is bind-mounted read-write into the container at the same absolute path (so task.json and artifacts land directly on the host filesystem). This makes --run-dir load-bearing for isolation:

  • Do not point --run-dir at a symlink. Docker resolves the source of a bind mount; following a symlink would silently grant the container RW access to a different host location.
  • Do not point --run-dir at a sensitive parent (e.g. $HOME directly, /etc, a repo root). Use a dedicated runs/ subtree.
  • The default (runs/<timestamp>/) is safe.

Boundary

Layer Location
Agent process (Claude Code SDK) inside container
Sandbox + per-row criterion checking inside container
task.json serialization container → host bind mount
Per-criterion aggregate() (P/R/F1, suite thresholds) host
Reports, run summary, experiment rollups host

task.json is the only artifact crossing the boundary. Aggregation reads it via the existing host pipeline unchanged.

Limitations

  • Relative template paths: template_sources[].path is resolved to a host absolute path before staging, so it won't exist inside the container unless you also forward the parent dir via sandbox.docker.extra_mounts.
  • No container reuse across tasks: each task = one fresh container. Adds ~1–3 s startup overhead per task; negligible vs. LLM latency.
  • macOS Keychain auth: not reachable from the container; set ANTHROPIC_API_KEY (direct) or Bedrock credentials instead.

Architecture

The host's DockerRunner (coder_eval/isolation/docker_runner.py) renders the docker run argv, bind-mounts task inputs at /work/input, allocates an output dir at /work/output, and tails container stdout into docker.log in the task's run dir.

If the task declares a reference: block, a throwaway copy of its directory is bind-mounted read-write at /work/references, and an empty tmpfs is layered over its original location inside the task-directory mount so the agent cannot reach the solution through $TASK_DIR. The mount is read-write on purpose — a :ro bind mount cannot be chmod'd (EROFS), and the orchestrator holds this path at mode 000 for the duration of each agent turn. The container also runs with --cap-drop DAC_OVERRIDE --cap-drop DAC_READ_SEARCH (see the caveat below for why FOWNER/CHOWN are deliberately kept).

What this does and does not guarantee. The mode-000 window stops an agent that passively reads the reference — ls, cat, grep all fail. Two things it does not stop:

  1. A deliberate re-chmod. chmod(2) is gated on owner-or-CAP_FOWNER, and the container runs as root, which owns the bind-mounted copy. Verified: chmod 755 /work/references succeeds from inside the container and the solution is then readable. Dropping FOWNER/CHOWN does not fix this and is deliberately not done — the in-container orchestrator that applies the window is the same root process with the same capabilities, so dropping FOWNER breaks the harness's own chmod wherever the bind mount preserves a non-root owner (native Linux). Verified: container root, uid-1000-owned directory, FOWNER dropped → chmod: Operation not permitted. The drop only bites on the hosts where it also disables the control.
  2. Waiting the window out. The window spans agent.communicate, so between turns and after the final one the reference is back at its normal mode, and nothing reaps the agent's child processes at turn end — a backgrounded read loop started during a turn succeeds once the window closes.

The write half of (2) IS closed: the reference tree is hashed at staging time and re-verified before grading, and a mismatch fails the run with ReferenceTamperedError rather than scoring reference_comparison against a file the agent wrote. Closing the read half fully requires running the agent as a non-root uid that does not own the reference, and holding the window for the agent's whole lifetime — tracked as follow-up. Treat the current control as raising the cost of accidental leakage, not as a boundary against an adversarial agent.

See Reference Solutions.

Inside the container, the entrypoint invokes coder-eval _run-task-internal (hidden subcommand), which loads the staged YAML + context, runs the standard in-process Orchestrator (driver auto-coerced back to tempdir), and writes task.json to the output mount. Host reads it and feeds the existing aggregation pipeline.

A result_kind discriminator on CriterionResult ensures ClassificationCriterionResult subclasses survive the JSON round-trip — without it, host-side aggregation would silently lose observed_label/expected_label.