Problem
runAgentWithRetry in evals/runner/experiment.ts replays a trial's whole prompt after a transient agent error (isTransientAgentError). The {{uniq}} marker introduced in #1423 is resolved once per trial in expandIterations, so the retry asks for the same schedule name. A create case that succeeded on the platform before the transient failure collides with itself on the retry.
#1423's guarantee covers concurrent runs and distinct trials, not a retry of the same trial. Confirmed during that PR's review as a pre-existing gap; the evals/README.md caveat about reading a failure describes it.
Proposed fix
Give each attempt its own suffix, e.g. -t<trial>r<attempt>, by resolving the marker inside the retry loop (or re-substituting from the unresolved item on retry) so a replayed prompt names a fresh resource. isNameFromRun's -<runId>-t match keeps working because the run id and -t delimiter are unchanged. Add a unit test that two attempts of one trial resolve to different names.
Context
Drafted by Claude Code during the #1394 shepherd run.
Problem
runAgentWithRetryinevals/runner/experiment.tsreplays a trial's whole prompt after a transient agent error (isTransientAgentError). The{{uniq}}marker introduced in #1423 is resolved once per trial inexpandIterations, so the retry asks for the same schedule name. A create case that succeeded on the platform before the transient failure collides with itself on the retry.#1423's guarantee covers concurrent runs and distinct trials, not a retry of the same trial. Confirmed during that PR's review as a pre-existing gap; the
evals/README.mdcaveat about reading a failure describes it.Proposed fix
Give each attempt its own suffix, e.g.
-t<trial>r<attempt>, by resolving the marker inside the retry loop (or re-substituting from the unresolved item on retry) so a replayed prompt names a fresh resource.isNameFromRun's-<runId>-tmatch keeps working because the run id and-tdelimiter are unchanged. Add a unit test that two attempts of one trial resolve to different names.Context
evals/run_id.ts,substituteUniqMarker).Drafted by Claude Code during the #1394 shepherd run.