Skip to content

release: retry the attestation download, and say why when it fails - #1317

Merged
justinjoy merged 1 commit into
mainfrom
fix/1290-attestation-retry
Sep 3, 2026
Merged

release: retry the attestation download, and say why when it fails#1317
justinjoy merged 1 commit into
mainfrom
fix/1290-attestation-retry

Conversation

@justinjoy

Copy link
Copy Markdown
Collaborator

Refs #1290. Stacked on #1289, which owns release-tag.yml.

The defect

The workflow attested the archive and downloaded the bundle in the next step, back to back, with no delay and no retry:

(cd dist && gh attestation download "wirelog-"*.tar.gz --repo "$GITHUB_REPOSITORY" --limit 1)
attestation=$(find dist -maxdepth 1 -type f -name 'sha256*.jsonl' -print -quit)
test -n "$attestation"

Those are separate API operations. Without read-your-writes consistency the download returns nothing and the job dies on a bare test -nafter every verification job has passed, on a tag that cannot be rebuilt.

Extracted, not just patched

The retry lives in scripts/release/download-attestation.sh so it can be exercised without cutting a release. That is the point rather than a side benefit: this is code whose first real execution is a release. Same reasoning #1293 established for generate-sbom.sh.

The failure path now diagnoses:

no attestation bundle for wirelog-9.9.9.tar.gz after 5 attempts
  The attest step and this download are separate API operations, so the
  usual cause is replication lag rather than a missing attestation.
  Check https://github.com/<repo>/attestations before re-running; if the
  attestation is listed there, re-running this job alone is enough.
  gh said: ...

Two problems the retry itself created

Both found in review, both regressions against the inline original:

problem why it exists
a stale bundle from an earlier run satisfied the existence check on attempt 1, and would be renamed and published as this release's provenance swallowing gh's status (needed so a transient failure doesn't abort before the loop runs) means success is decided by "a bundle exists", not "this download produced one". The inline form couldn't do this — gh's exit aborted the step
a zero/non-numeric attempt count ran the loop zero times, never called gh, and still printed the full replication-lag diagnosis a confidently wrong answer

Both guarded and pinned. Currently unreachable in CI (fresh VM per job), but the guard is now the code rather than runner ephemerality.

Also fixed in review: an assertion that gave false credit for the diagnosis text — the needle attestations was satisfied by the stub's own stderr (no attestations found for subject), so deleting the URL and re-run guidance left the suite green. It now matches text only the script can produce.

Validation

  • 23 assertions, suite abi. Confirmed by the reviewer to actually run in release-tag.yml's ABI job and ci-pr.yml — needs only bash and a stubbed gh, no b3sum. (I have shipped two tests this session that skipped in 100% of CI; this one was checked.)
  • Mutations killed: removing the retry (the original bug), removing the stale-bundle guard, removing the attempts validation, removing the URL guidance, cp instead of mv, dropping the gh quote, leaving the log behind
  • The test asserts it actually retried, by counting stub invocations, rather than inferring it from success
  • No pipelines in the new script, so the pipefail early-exit class this repo has hit seven times is structurally absent
  • Full suite 306 Ok / 0 Fail / 12 Skipped

One redirection-order bug of mine was caught by the suite itself: 2>/dev/null >&2 sends stdout to /dev/null, swallowing the gh output that line exists to print.

@justinjoy
justinjoy force-pushed the fix/750-signing-and-publication branch from 388f3d2 to 6fef16f Compare September 2, 2026 12:53
@justinjoy
justinjoy force-pushed the fix/750-signing-and-publication branch from 6fef16f to 5492454 Compare September 2, 2026 13:07
Base automatically changed from fix/750-signing-and-publication to main September 2, 2026 16:46
@justinjoy
justinjoy force-pushed the fix/1290-attestation-retry branch from 7edcfd0 to 5109979 Compare September 3, 2026 03:25
release-tag.yml attested the source archive and downloaded the provenance
bundle in the next step, back to back, with no delay and no retry.  Those
are separate API operations, so if the attestations API is not
read-your-writes consistent the download returns nothing -- and the job
died on a bare `test -n` with no indication of why, after every
verification job had already passed, on a tag that cannot be rebuilt.

The download now retries a bounded number of times and, on final failure,
names replication lag as the usual cause, points at the repository's
attestations page, says that re-running this job alone is enough, and
quotes what gh actually said.

It lives in a script rather than inline in the workflow so it can be
exercised without cutting a release.  That is the whole point: the path
it guards runs for the first time on a real one.

Two things the retry made necessary that the inline version did not need.
Swallowing gh's status so a transient failure does not abort before the
loop has had its say means success is decided by "a bundle exists" rather
than "this download produced one" -- so a bundle left by an earlier run
would be renamed and published as this release's provenance after a
single failed call.  The inline form could not do that, because gh's exit
aborted the step.  The directory is now cleared before the loop.  And a
zero or non-numeric attempt count would run the loop zero times, never
call gh, and still print the full diagnosis; it is rejected instead.

Refs #1290.
@justinjoy
justinjoy force-pushed the fix/1290-attestation-retry branch from 5109979 to ac717e0 Compare September 3, 2026 09:24
@justinjoy
justinjoy merged commit 96c168d into main Sep 3, 2026
26 checks passed
@justinjoy
justinjoy deleted the fix/1290-attestation-retry branch September 3, 2026 12:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant