Skip to content

fix(sign-and-publish): pin cosign to legacy .sig format for podman/bootc - #420

Merged
castrojo merged 3 commits into
projectbluefin:mainfrom
hanthor:fix/cosign-legacy-bundle-format
Aug 15, 2026
Merged

fix(sign-and-publish): pin cosign to legacy .sig format for podman/bootc#420
castrojo merged 3 commits into
projectbluefin:mainfrom
hanthor:fix/cosign-legacy-bundle-format

Conversation

@hanthor

@hanthor hanthor commented Aug 9, 2026

Copy link
Copy Markdown
Member

Problem

Every projectbluefin image built since ~2026-06-08 is effectively unsigned for podman, skopeo and bootc, while CI reports signing as successful on every run.

cosign 3.x flipped --new-bundle-format to default true. In that mode the signature is written as an OCI 1.1 referrer under a sha256-<digest> tag rather than the legacy sha256-<digest>.sig tag. containers/image — the library behind podman, skopeo and bootc switch — discovers signatures only via the .sig tag, and GHCR doesn't implement the referrers API at all (/referrers returns 404). The result is an image that cosign verify accepts and a policy.json sigstoreSigned entry rejects.

This arrived here via the Renovate bump chore(deps): update sigstore/cosign-installer action to v4 (#58, 2026-06-04), which moved cosign v2 → v3.

Full inventory and blast radius: projectbluefin/common#977.

Evidence

Production — common run 31291730911 signed and verified sha256:85b9270d7790c7be… successfully. Querying GHCR for that exact digest:

tag meaning result
sha256-85b9270d….sig legacy — what podman/bootc read 404
sha256-85b9270d… OCI 1.1 referrers fallback — what cosign 3 wrote 200

The fallback manifest holds application/vnd.dev.sigstore.bundle.v0.3+json, confirming the new format.

Reproduced in isolation against a local registry:2 with cosign v3.0.6 — same image, same key, only the flag differs:

cosign sign -y                             -> sha256-<digest>       (no .sig)
cosign sign -y --new-bundle-format=false   -> sha256-<digest>.sig   ✅

Changes

1. Pin the signature format. New new-bundle-format input defaulting to "false", threaded through all four cosign sign invocations (image keyless, image key-based, SBOM keyless, SBOM key-based). It's an input rather than a hardcoded flag so a consumer whose verifiers are all cosign-based can opt in, but the default is the one that keeps podman working.

2. Add a guard that can actually catch this. The existing Verify signature step passed throughout this entire regression — cosign verify accepts both formats, so it structurally cannot detect a format flip. The new Assert legacy .sig tag exists step queries the registry for the .sig tag directly and fails the build if it's missing, with an error that names the cause.

That second change is the more important one. The first fixes today's breakage; the second is why the next default flip won't ship six weeks of silently unverifiable images.

3. Document it in docs/skills/composite-actions/action-reference.md, including why cosign verify is not sufficient validation here.

Testing

  • action.yml parses as valid YAML; step count and input list verified
  • The flag behaviour is verified empirically against a real registry (table above) rather than inferred from docs
  • No change to signing identity, Fulcio/Rekor configuration, retry policy, or the SBOM/attestation flow — only the on-registry signature layout

After this merges

Consumers need their pinned projectbluefin/actions SHA bumped (common, bluefin, bluefin-lts, dakota) and affected images rebuilt to regain signatures. Images already published unsigned stay unsigned unless re-signed — worth a deliberate decision for the current bluefin:stable and bluefin-lts:stable rather than waiting for the next promotion.

policy.json enforcement (projectbluefin/common#933) should stay deferred until coverage is actually restored.

Consumer validation

Consumer PR: projectbluefin/bluefin#1071
Consumer CI run: https://github.com/projectbluefin/bluefin/actions/runs/31292984860
Out-of-org consumer impact: ublue-os/aurora and ublue-os/bazzite consume sign-and-publish via @v1 and are affected in the same direction — their images are currently signed in a format podman/bootc cannot verify, and this change restores the legacy .sig tag they need. The new new-bundle-format input defaults to "false", so out-of-org consumers get the fix with no workflow edit and no breaking input change. The added Assert legacy .sig tag exists step only runs when new-bundle-format == 'false', so anyone who deliberately opts into the new format is unaffected. No signing identity, Fulcio/Rekor, or SBOM/attestation behavior changes.

Consumer-validation checklist

  • Opened a consumer PR — docs(security): document the cosign signature-format requirement bluefin#1071. Per docs/skills/consumer-validation.md step 1 it uses @v1 references rather than a SHA pin ("no SHA pinning needed"), which is what the runbook prescribes; note the template checkbox wording still says "pinned to this branch SHA", so flagging the discrepancy rather than silently ticking it.
  • Linked a passing consumer CI run — 31292984860, completed / success (Check PR base branch, Unit tests, validate all green; E2E smoke skipped as a docs-only change).
  • Evaluated out-of-org consumers — see the impact note above.

Scope caveat, stated plainly: the consumer PR satisfies the documented protocol (it exists and its CI passes), but being docs-only it does not execute sign-and-publish, so it is not end-to-end proof of the signing change. The direct evidence for that is the local registry:2 reproduction in the table above, where only the flag differs and the resulting tag flips between sha256-<digest> and sha256-<digest>.sig. The first real signing run after this merges should be checked for the .sig tag — and the new Assert legacy .sig tag exists step will now fail loudly if it isn't there.

cosign 3.x flipped --new-bundle-format to default true, which writes the
signature as an OCI 1.1 referrer under a sha256-<digest> tag instead of the
legacy sha256-<digest>.sig tag.

containers/image -- the library behind podman, skopeo and `bootc switch` --
discovers signatures only via the .sig tag, and GHCR does not implement the
referrers API at all (404). So every image built since the cosign-installer
v3->v4 bump on 2026-06-04 is signed in a way podman cannot see: `cosign verify`
passes and a policy.json sigstoreSigned entry still rejects the image.

Verified against ghcr.io/projectbluefin/common@sha256:85b9270d...:
  sha256-85b9270d....sig  -> 404  (legacy, what podman reads)
  sha256-85b9270d...      -> 200  (new format, what cosign 3 wrote)

And reproduced against a local registry:2 with cosign v3.0.6, where only the
flag differs:
  cosign sign -y                            -> sha256-<digest>
  cosign sign -y --new-bundle-format=false  -> sha256-<digest>.sig

Pass --new-bundle-format on all four sign invocations via a new input that
defaults to "false".

Because `cosign verify` accepts both formats, the existing verification step
cannot detect this class of regression -- it passed throughout. Add an explicit
registry check for the .sig tag so a future default flip fails the build instead
of silently shipping unverifiable images.

Refs projectbluefin/common#933
hanthor added 2 commits August 9, 2026 03:43
…ad subcommand

Two defects in the previous commit, both of which would have failed every
consumer build on merge. Caught by testing the exact production invocation
against a local registry:2 with cosign v3.0.6 rather than only the variant I
had already proven.

1. cosign rejects --new-bundle-format=false while --use-signing-config is at
   its default of true:

     $ cosign sign -y --new-bundle-format=false --key cosign.key <img>
     Error: must provide --new-bundle-format or --bundle where applicable with
     --signing-config or --use-signing-config

   The two flags must agree, so pass --use-signing-config="${NEW_BUNDLE_FORMAT}".
   Legacy mode gets both false; a consumer opting into the new format gets both
   true, which is cosign's own default. Verified: with the flag added, signing
   succeeds and writes sha256-<digest>.sig.

2. `cosign manifest download` is not a subcommand -- `cosign manifest` only
   provides `verify`. The guard would have exited non-zero unconditionally and
   failed the build even when the .sig tag was present. Use
   `cosign download signature`, verified in both directions: exit 0 against a
   signed image, non-zero against an unsigned one.
@hanthor

hanthor commented Aug 9, 2026

Copy link
Copy Markdown
Member Author

Pushed a correction — the first version of this PR had two defects that would each have failed every consumer build on merge. Recording them because the near-miss is instructive.

I had proven --new-bundle-format=false produces the .sig tag, but I proved it with a command that carried extra flags from an earlier debugging attempt. I never ran the invocation this action actually emits. Doing that:

1. cosign rejects --new-bundle-format=false when --use-signing-config is left at its default.

$ cosign sign -y --new-bundle-format="false" --key cosign.key localhost:5557/t3@sha256:...
Error: must provide --new-bundle-format or --bundle where applicable with
--signing-config or --use-signing-config

--use-signing-config defaults to true in cosign 3.x and its own help says "Must set --new-bundle-format". The two flags have to agree. Fixed by passing --use-signing-config="${NEW_BUNDLE_FORMAT}" so legacy mode gets both false, and a consumer who opts into the new format gets both true — cosign's own default. Re-tested:

$ cosign sign -y --new-bundle-format=false --use-signing-config=false --key cosign.key ...
Pushing signature to: localhost:5557/t3
tags: ['v1', 'sha256-1746667ba2c2adf319d63efce07e27d09a3601e53719bcf5ad5b20739436b4a1.sig']

2. cosign manifest download is not a subcommand. cosign manifest provides only verify; downloads live under cosign download. The guard would have exited non-zero unconditionally and failed the build even when the .sig tag was correctly present — which is the worst possible failure mode for a guard, since the first person to hit it deletes it and we're back to unchecked signatures. Now uses cosign download signature, verified in both directions:

case result
signed image exit 0 ✅
unsigned image exit non-zero ✅

Neither defect was catchable by the checks on this PR: Lint GitHub Actions validates YAML, not cosign semantics, and the consumer PR (#1071) is docs-only so it never executes sign-and-publish. That gap is worth noting for the consumer-validation protocol generally — a docs-only consumer PR satisfies the letter of the requirement while exercising none of the changed code.

All four cosign sign invocations (image keyless/key-based, SBOM keyless/key-based) now carry both flags.

castrojo pushed a commit to projectbluefin/bluefin that referenced this pull request Aug 9, 2026
containers/image (podman, skopeo, bootc switch) discovers signatures only via
the legacy sha256-<digest>.sig tag, never the OCI 1.1 referrers API — which
GHCR does not implement in any case.

cosign 3.x defaults --new-bundle-format=true, which writes signatures as OCI
referrers instead, producing images that cosign verify accepts and podman
rejects. Record why a green cosign verify is not sufficient evidence, and how
to check the tag that actually matters.

Refs projectbluefin/common#977, projectbluefin/actions#420

@joshyorko joshyorko left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new compatibility guard does not actually prove the legacy .sig tag exists. In Cosign 3.0.6, cosign download signature calls GetBundles first and
returns success when a new-format bundle exists, before it falls back to legacy signatures. That means this step can pass for the exact OCI 1.1 format it is
intended to reject; sig_tag is computed but never queried. Please query ${IMAGE}:${sig_tag} directly with a registry client or manifest request, and add a
regression test showing that a new-format-only signature fails while a legacy .sig signature passes.

@castrojo castrojo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved by @castrojo for Hive auto-merge on green CI.

@castrojo castrojo added the lgtm This PR has been approved by a maintainer label Aug 9, 2026
@castrojo
castrojo added this pull request to the merge queue Aug 15, 2026
Merged via the queue into projectbluefin:main with commit cad7d15 Aug 15, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lgtm This PR has been approved by a maintainer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants