fix(sec): ship Sigstore trust materials (Fulcio root + Rekor key)

[hanthor]

Summary

• Adds a sigstoreSigned entry for ghcr.io/projectbluefin to system_files/shared/etc/containers/policy.json enforcing keyless OIDC signature verification for container images built/published via GitHub Actions OIDC keyless signing.
• Adds fulcio_v1.crt.pem and rekor.pub Sigstore root assets to system_files/shared/usr/lib/pki/containers/.

Closes #847

[castrojo]

This doesn't work with bootc iirc? It just gracefully falls back?

[castrojo]

Deferred to the human queue during backlog triage. This is a signature-enforcement change on system_files/shared/, so it needs a deliberate security review rather than a fast triage pass. Recording partial findings so the next reviewer starts ahead.

What it does: adds a sigstoreSigned policy entry scoped to ghcr.io/projectbluefin, plus the Fulcio root cert and Rekor public key it depends on. This is enforcement, not advisory — the default in policy.json is reject.

Verified so far:

• Signing is keyless OIDC via projectbluefin/actions/bootc-build/sign-and-publish (build.yml:200-208).
• The workflow's own verification regex is ^https://github\.com/projectbluefin/(common|actions)/\.github/workflows/. The regex proposed here is a superset that also covers bluefin, bluefin-lts, dakota, aurora, so it is consistent with how common signs today.
• The existing ghcr.io/ublue-os entry uses keyPaths with long-lived public keys, whereas this uses the Fulcio/Rekor keyless path. Both are valid, they are just different trust models living side by side.

Open questions a reviewer should settle before merging:

1. Namespace coverage. The scope key ghcr.io/projectbluefin applies to every repository under that namespace, not only the variant images. Any auxiliary or older image under ghcr.io/projectbluefin that was never cosign-signed with a matching identity becomes unpullable once this lands. Worth enumerating the namespace and confirming each published image verifies.
2. Identity subject for reusable workflows. sign-and-publish is a composite action, which does not change the OIDC subject, but if any variant signs through a reusable workflow the certificate subject becomes the job_workflow_ref of that workflow. That would need to be inside the regex.
3. Downstream variants. shared/ reaches bluefin, bluefin-lts, dakota, and Aurora. Each needs its signing identity confirmed against this regex, not just common.
4. Rollback story. If verification fails post-merge, affected systems cannot bootc upgrade their way out, since the pull itself is refused. Worth knowing the recovery path before enabling.

Minor: fulcio_v1.crt.pem has no trailing newline (\ No newline at end of file). Go's pem.Decode tolerates this, so it is cosmetic, but adding the newline costs nothing.

Suggested verification before merge:

# Enumerate everything the new scope would govern
gh api /orgs/projectbluefin/packages?package_type=container --jq '.[].name'

# Confirm a real image verifies against the proposed identity
cosign verify ghcr.io/projectbluefin/common:latest \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  --certificate-identity-regexp '^https://github\.com/projectbluefin/(bluefin|bluefin-lts|dakota|common|aurora|actions)/\.github/workflows/'

No objection to the approach — the gap it closes is real, since ghcr.io/projectbluefin currently falls through to insecureAcceptAnything. It just should not land on a triage pass.

[kubestellar-hive]

[strategist] CI note for the human reviewer: the validate failure on the current head (ec46431) is not the policy change — it is end-of-file-fixer wanting a trailing newline in system_files/shared/usr/lib/pki/containers/fulcio_v1.crt.pem (pre-commit run 31137512568). One-line fix; everything else (builds, trivy, compose test) is green.

---

🐝 Hive Agent: strategist | Instance: hosted-projectbluefin-knuckle-gjvq

[castrojo]

The entry matches the #847 recommendation verbatim and the materials check out: policy.json remains valid JSON, the Fulcio cert is the genuine Sigstore production root (self-signed O=sigstore.dev, CN=sigstore, valid 2021-10-07 → 2031-10-05), and rekor.pub is the standard public-good Rekor key. matchRepository + the GitHub Actions OIDC issuer + workflow-path subjectRegExp is the right shape for our keyless cosign signing.

Two blocking issues:

1. CI is red (pre-commit). end-of-file-fixer fails on system_files/shared/usr/lib/pki/containers/fulcio_v1.crt.pem — missing trailing newline. One-character fix.

2. Namespace-wide scope will break pulls of unsigned/differently-signed images (Breakage gate). The entry covers all of ghcr.io/projectbluefin with sigstoreSigned, which replaces the insecureAcceptAnything catch-all for every image in the namespace — not just bootc OS images. The org publishes many packages that are not signed by the listed workflows, e.g. bluefin-toolbox / ubuntu-toolbox (commonly pulled via distrobox/toolbox on Bluefin itself), finpilot, knuckle, testsuite, brew, lab-runner, bluefin-cache, base/static/skopeo/buildah CI images — and crucially the unsigned bluefin-pr-* PR test images, since "Create and sign multi-arch manifest" is skipped on PR builds. After this lands, bootc switch to a PR test image from a Bluefin machine would be rejected, breaking the factory's own test loop.

Recommend scoping to the signed OS image repos using policy.json's longest-prefix matching, e.g. separate entries for ghcr.io/projectbluefin/bluefin, .../bluefin-lts, .../dakota, .../common (and a deliberate decision on bluefin-pr-*), or first confirming every published package is signed with a matching identity. Minor: aurora in the subjectRegExp is dead weight — projectbluefin/aurora doesn't exist.

Happy to re-approve quickly once the scope is narrowed or an inventory shows full signing coverage.

[hanthor]

@castrojo thanks — point 1 fixed, and point 2 turned out to be worse than either of us thought, so I've split the PR.

What this PR is now: trust materials only (fulcio_v1.crt.pem + rekor.pub, trailing newline fixed). policy.json is reverted to match main — zero enforcement, zero behaviour change. That should clear end-of-file-fixer and makes this safely mergeable on its own, so the enforcement entry can land later without a second materials rollout.

Why the policy entry is deferred: I inventoried actual signing coverage in ghcr.io/projectbluefin before narrowing the scope, and the premise of the narrowing doesn't hold. GHCR doesn't implement the OCI referrers API (404 across the board), so cosign's sha256-<digest>.sig tag is the only mechanism — I checked both the index digest and the per-arch child manifests for each tag:

image:tag	index sig	per-arch sig
bluefin:stable	❌	— (single manifest)
bluefin:gts	❌	—
bluefin:testing	❌	—
bluefin:stable-daily	✅
bluefin:lts / bluefin:10	✅
bluefin-lts:stable	❌	❌ amd64, ❌ arm64
bluefin-lts:testing	❌	❌ amd64, ❌ arm64
bluefin-lts-nvidia:stable, bluefin-lts-hwe:*	❌
dakota:stable	❌
dakota:latest	✅
dakota-nvidia:*, dakota-gaming:*	❌
common:latest	❌	❌ amd64, ❌ arm64
bluefin-toolbox:latest	✅
ubuntu-toolbox:latest	✅
brew:latest	✅
lab-runner, base, static, skopeo, buildah, python, ramalama, qemu-img, finpilot:*	❌

Two consequences:

1. The narrowing you suggested — separate entries for bluefin, bluefin-lts, dakota, common — would break more than the namespace-wide entry does for real users. bluefin:stable and bluefin:gts are unsigned, and those are the tags most people are actually on; every bluefin-lts tag is unsigned; dakota:stable and common:latest are unsigned. Enforcing on exactly those four repos rejects the flagship update path.
2. The toolbox images are the opposite of the concern. bluefin-toolbox (216/217 tags), ubuntu-toolbox and brew are consistently signed — they're among the few things in the namespace that would pass.

The pattern that stands out: within a single repo, the daily/build tags are signed (bluefin:stable-daily, dakota:latest) while the promoted tags are not (bluefin:stable, bluefin:gts, dakota:stable). That looks like the promotion path re-pushes or re-tags the manifest without carrying or re-creating the signature, which is a signing-coverage bug in its own right rather than anything policy.json can paper over.

So I don't think there's a correct scope to pick here yet — enforcement should follow signing coverage, not lead it. Suggested sequence:

1. Merge this PR (materials only, no enforcement).
2. Fix the promotion path so stable/gts/lts inherit or regain signatures, and decide whether bluefin-lts gets signed at all.
3. Re-add the policy.json entry scoped to whatever is verifiably signed at that point, with a deliberate call on the unsigned bluefin-pr-* test images (they'd block bootc switch to PR images from a Bluefin machine).

Happy to open issues for (2) if you agree with the read. Also noted re: aurora being dead weight — it's gone along with the rest of the entry, and I'll leave it out when the entry comes back.

[hanthor]

Correction / sharpening of my inventory above — I pulled build timestamps from the config blobs, and the pattern isn't the promotion path. It's a date cutoff:

image:tag	built	signed
brew:latest	2026-02-15	✅
bluefin:lts	2026-06-02	✅
bluefin:stable-daily	2026-06-06	✅
dakota:latest	2026-06-08	✅
bluefin:stable	2026-07-20	❌
dakota:stable	2026-07-30	❌
bluefin-lts:stable	2026-08-03	❌
bluefin:testing	2026-08-04	❌
common:latest	2026-08-09	❌

(bluefin-toolbox:latest is signed but dates to 2024, so it's consistent with the old regime rather than evidence against the cutoff.)

Every image built on or before ~2026-06-08 is signed; every image built after mid-June 2026 is unsigned, across all repos and both promoted and daily tags. So my earlier read — "the promotion path drops signatures" — was wrong. The signed tags just happen to be the stale ones that predate the break; bluefin:stable-daily looked healthy only because it hasn't rebuilt since June.

That makes this a single org-wide signing regression around mid-June 2026, not a per-repo or per-tag issue — one root cause to find rather than a promotion redesign. It also means signing coverage will keep decaying: each rebuild converts a signed tag into an unsigned one, which is why bluefin:stable went dark when it was last promoted on 2026-07-20.

Doesn't change the recommendation — this PR should still land as materials-only and enforcement should follow coverage — but it makes step 2 much more tractable. Worth checking what changed in the cosign/signing step around 2026-06-08; if you want I can open an issue with this table.

[castrojo]

Assessment (agent follow-up on the request-changes review)

Both blocking issues from the review are addressed at HEAD (1b523fb):

1. CI / EOF newline — fixed; pre-commit and all checks are green on the latest run.
2. Namespace-wide scope — the policy.json change has been removed entirely; the PR now ships only the Sigstore trust materials (fulcio_v1.crt.pem, rekor.pub) and defers enforcement.

Independent verification of the signing inventory

I spot-checked ghcr.io/projectbluefin/bluefin directly against the registry (anonymous pull token), and the picture is slightly more nuanced than "nothing is signed":

• stable-daily and latest-20260530: the index digest has a sha256-<digest>.sig tag. Decoding the Fulcio cert in the bundle shows a keyless signature with SAN https://github.com/projectbluefin/bluefin/.github/workflows/reusable-build.yml@refs/heads/stable.
• testing: the index digest has no .sig tag — unsigned.
• PR builds: Create and sign multi-arch manifest in common's build.yml is gated on github.event_name != 'pull_request' && github.event_name != 'merge_group', so PR test images are unsigned by construction.

Two consequences for the future enforcement PR:

1. Any sigstoreSigned entry today — namespace-wide or narrowed to ghcr.io/projectbluefin/bluefin — would hard-reject bluefin:testing and PR test images, breaking the factory test loop (plus toolboxes/CI images in the namespace). The deferral here is the right call.
2. The original entry's subjectRegExp pointed at projectbluefin/(common|actions) workflows, but the OS images that are signed carry the bluefin repo's reusable-build.yml identity. Enforcement will need per-repo entries with each repo's own workflow identity (e.g. bluefin/.../reusable-build.yml, the bluefin-lts/dakota equivalents), and a deliberate decision on how to handle testing/PR images — either sign them too, or leave their scopes out of enforcement.

Proposed direction

• Land this PR as-is (materials only). It's a no-op for policy behavior and removes the need for a second materials rollout later.
• Follow-up work for [sec-check] policy.json: ghcr.io/projectbluefin missing sigstoreSigned entry — images accepted without signature verification #847: per-repo enforcement entries once testing and PR-image signing is decided, with subjectRegExp matching each repo's actual signing workflow.

Note: I can't re-review or dismiss the outstanding CHANGES_REQUESTED — that needs a human re-review (cc @castrojo). CI is green and the branch is mergeable pending that.

[hanthor]

Root cause found and fixed — deferring the policy entry here was the right call, and the reason is now concrete.

cosign 3.x is why nothing is signed. The Renovate bump of sigstore/cosign-installer v3→v4 in projectbluefin/actions on 2026-06-04 moved us to cosign v3, which flipped --new-bundle-format to default true. That writes signatures as OCI 1.1 referrers under a sha256-<digest> tag instead of the legacy sha256-<digest>.sig tag. containers/image — podman, skopeo, bootc switch — reads only the .sig tag, so a sigstoreSigned entry cannot see them. cosign verify passes throughout, which is why CI never noticed.

Confirmed against this repo's own latest build (common@sha256:85b9270d…): .sig tag → 404, bare sha256-… referrers tag → 200, containing application/vnd.dev.sigstore.bundle.v0.3+json.

That also corrects my table above: the clean date cutoff isn't the promotion path dropping signatures, it's the cosign upgrade. Anything that still looks signed simply hasn't rebuilt since early June.

• Issue with full inventory and blast radius: Container signing broken since ~2026-06-08: cosign 3.x writes new-format bundles that podman/bootc cannot see #977
• Fix: fix(sign-and-publish): pin cosign to legacy .sig format for podman/bootc actions#420 — --new-bundle-format=false on all four sign invocations, plus a registry assertion for the .sig tag, since the existing cosign verify step is structurally incapable of catching this (it accepts both formats)

Once that lands, consumers bump their pinned actions SHA and rebuild, and signing coverage returns — at which point the policy.json entry can be scoped to what's verifiably signed, which was your original point on the Breakage gate.