Skip to content

fix(evals): Re-resolve {{uniq}} per retry attempt #1426

Description

@jirispilka

Problem

runAgentWithRetry in evals/runner/experiment.ts replays a trial's whole prompt after a transient agent error (isTransientAgentError). The {{uniq}} marker introduced in #1423 is resolved once per trial in expandIterations, so the retry asks for the same schedule name. A create case that succeeded on the platform before the transient failure collides with itself on the retry.

#1423's guarantee covers concurrent runs and distinct trials, not a retry of the same trial. Confirmed during that PR's review as a pre-existing gap; the evals/README.md caveat about reading a failure describes it.

Proposed fix

Give each attempt its own suffix, e.g. -t<trial>r<attempt>, by resolving the marker inside the retry loop (or re-substituting from the unresolved item on retry) so a replayed prompt names a fresh resource. isNameFromRun's -<runId>-t match keeps working because the run id and -t delimiter are unchanged. Add a unit test that two attempts of one trial resolve to different names.

Context

Drafted by Claude Code during the #1394 shepherd run.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    t-aiIssues owned by the AI team.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions