fix(sign-and-publish): pin cosign to legacy .sig format for podman/bootc - #420
Conversation
cosign 3.x flipped --new-bundle-format to default true, which writes the signature as an OCI 1.1 referrer under a sha256-<digest> tag instead of the legacy sha256-<digest>.sig tag. containers/image -- the library behind podman, skopeo and `bootc switch` -- discovers signatures only via the .sig tag, and GHCR does not implement the referrers API at all (404). So every image built since the cosign-installer v3->v4 bump on 2026-06-04 is signed in a way podman cannot see: `cosign verify` passes and a policy.json sigstoreSigned entry still rejects the image. Verified against ghcr.io/projectbluefin/common@sha256:85b9270d...: sha256-85b9270d....sig -> 404 (legacy, what podman reads) sha256-85b9270d... -> 200 (new format, what cosign 3 wrote) And reproduced against a local registry:2 with cosign v3.0.6, where only the flag differs: cosign sign -y -> sha256-<digest> cosign sign -y --new-bundle-format=false -> sha256-<digest>.sig Pass --new-bundle-format on all four sign invocations via a new input that defaults to "false". Because `cosign verify` accepts both formats, the existing verification step cannot detect this class of regression -- it passed throughout. Add an explicit registry check for the .sig tag so a future default flip fails the build instead of silently shipping unverifiable images. Refs projectbluefin/common#933
…ad subcommand
Two defects in the previous commit, both of which would have failed every
consumer build on merge. Caught by testing the exact production invocation
against a local registry:2 with cosign v3.0.6 rather than only the variant I
had already proven.
1. cosign rejects --new-bundle-format=false while --use-signing-config is at
its default of true:
$ cosign sign -y --new-bundle-format=false --key cosign.key <img>
Error: must provide --new-bundle-format or --bundle where applicable with
--signing-config or --use-signing-config
The two flags must agree, so pass --use-signing-config="${NEW_BUNDLE_FORMAT}".
Legacy mode gets both false; a consumer opting into the new format gets both
true, which is cosign's own default. Verified: with the flag added, signing
succeeds and writes sha256-<digest>.sig.
2. `cosign manifest download` is not a subcommand -- `cosign manifest` only
provides `verify`. The guard would have exited non-zero unconditionally and
failed the build even when the .sig tag was present. Use
`cosign download signature`, verified in both directions: exit 0 against a
signed image, non-zero against an unsigned one.
|
Pushed a correction — the first version of this PR had two defects that would each have failed every consumer build on merge. Recording them because the near-miss is instructive. I had proven 1. cosign rejects
2.
Neither defect was catchable by the checks on this PR: All four |
containers/image (podman, skopeo, bootc switch) discovers signatures only via the legacy sha256-<digest>.sig tag, never the OCI 1.1 referrers API — which GHCR does not implement in any case. cosign 3.x defaults --new-bundle-format=true, which writes signatures as OCI referrers instead, producing images that cosign verify accepts and podman rejects. Record why a green cosign verify is not sufficient evidence, and how to check the tag that actually matters. Refs projectbluefin/common#977, projectbluefin/actions#420
joshyorko
left a comment
There was a problem hiding this comment.
The new compatibility guard does not actually prove the legacy .sig tag exists. In Cosign 3.0.6, cosign download signature calls GetBundles first and
returns success when a new-format bundle exists, before it falls back to legacy signatures. That means this step can pass for the exact OCI 1.1 format it is
intended to reject; sig_tag is computed but never queried. Please query ${IMAGE}:${sig_tag} directly with a registry client or manifest request, and add a
regression test showing that a new-format-only signature fails while a legacy .sig signature passes.
Problem
Every projectbluefin image built since ~2026-06-08 is effectively unsigned for podman, skopeo and
bootc, while CI reports signing as successful on every run.cosign 3.x flipped
--new-bundle-formatto defaulttrue. In that mode the signature is written as an OCI 1.1 referrer under asha256-<digest>tag rather than the legacysha256-<digest>.sigtag.containers/image— the library behind podman, skopeo andbootc switch— discovers signatures only via the.sigtag, and GHCR doesn't implement the referrers API at all (/referrersreturns 404). The result is an image thatcosign verifyaccepts and apolicy.jsonsigstoreSignedentry rejects.This arrived here via the Renovate bump
chore(deps): update sigstore/cosign-installer action to v4(#58, 2026-06-04), which moved cosign v2 → v3.Full inventory and blast radius: projectbluefin/common#977.
Evidence
Production —
commonrun 31291730911 signed and verifiedsha256:85b9270d7790c7be…successfully. Querying GHCR for that exact digest:sha256-85b9270d….sigsha256-85b9270d…The fallback manifest holds
application/vnd.dev.sigstore.bundle.v0.3+json, confirming the new format.Reproduced in isolation against a local
registry:2with cosign v3.0.6 — same image, same key, only the flag differs:Changes
1. Pin the signature format. New
new-bundle-formatinput defaulting to"false", threaded through all fourcosign signinvocations (image keyless, image key-based, SBOM keyless, SBOM key-based). It's an input rather than a hardcoded flag so a consumer whose verifiers are all cosign-based can opt in, but the default is the one that keeps podman working.2. Add a guard that can actually catch this. The existing
Verify signaturestep passed throughout this entire regression —cosign verifyaccepts both formats, so it structurally cannot detect a format flip. The newAssert legacy .sig tag existsstep queries the registry for the.sigtag directly and fails the build if it's missing, with an error that names the cause.That second change is the more important one. The first fixes today's breakage; the second is why the next default flip won't ship six weeks of silently unverifiable images.
3. Document it in
docs/skills/composite-actions/action-reference.md, including whycosign verifyis not sufficient validation here.Testing
action.ymlparses as valid YAML; step count and input list verifiedAfter this merges
Consumers need their pinned
projectbluefin/actionsSHA bumped (common,bluefin,bluefin-lts,dakota) and affected images rebuilt to regain signatures. Images already published unsigned stay unsigned unless re-signed — worth a deliberate decision for the currentbluefin:stableandbluefin-lts:stablerather than waiting for the next promotion.policy.jsonenforcement (projectbluefin/common#933) should stay deferred until coverage is actually restored.Consumer validation
Consumer PR: projectbluefin/bluefin#1071
Consumer CI run: https://github.com/projectbluefin/bluefin/actions/runs/31292984860
Out-of-org consumer impact:
ublue-os/auroraandublue-os/bazziteconsumesign-and-publishvia@v1and are affected in the same direction — their images are currently signed in a format podman/bootc cannot verify, and this change restores the legacy.sigtag they need. The newnew-bundle-formatinput defaults to"false", so out-of-org consumers get the fix with no workflow edit and no breaking input change. The addedAssert legacy .sig tag existsstep only runs whennew-bundle-format == 'false', so anyone who deliberately opts into the new format is unaffected. No signing identity, Fulcio/Rekor, or SBOM/attestation behavior changes.Consumer-validation checklist
docs/skills/consumer-validation.mdstep 1 it uses@v1references rather than a SHA pin ("no SHA pinning needed"), which is what the runbook prescribes; note the template checkbox wording still says "pinned to this branch SHA", so flagging the discrepancy rather than silently ticking it.completed / success(Check PR base branch,Unit tests,validateall green;E2E smokeskipped as a docs-only change).Scope caveat, stated plainly: the consumer PR satisfies the documented protocol (it exists and its CI passes), but being docs-only it does not execute
sign-and-publish, so it is not end-to-end proof of the signing change. The direct evidence for that is the localregistry:2reproduction in the table above, where only the flag differs and the resulting tag flips betweensha256-<digest>andsha256-<digest>.sig. The first real signing run after this merges should be checked for the.sigtag — and the newAssert legacy .sig tag existsstep will now fail loudly if it isn't there.