Skip to content

docs: state which images CertificateAudit can assess #175

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
stevebeattie merged 3 commits into from
Sep 10, 2026
Merged

docs: state which images CertificateAudit can assess #175

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
f059fb4
docs: state which images CertificateAudit can assess
stevebeattie Sep 3, 2026
24981fc
Correct the triage guidance and reconcile the /kaniko facts
stevebeattie Sep 9, 2026
91c7822
Verify the /kaniko claims against the image, and fix the auth diagnostic
stevebeattie Sep 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
162 changes: 151 additions & 11 deletions docs/certificate-audit.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,99 @@ apko's own comment on `writeCABundleChecksums` states the purpose: "so
downstream tooling (e.g. OpenSCAP) can verify they were not modified
post-build." This rule is that downstream tooling.

### Which images this rule can assess

Because the expected digest comes from the sidecar, an image that has no sidecar
has nothing to compare against — and the rule **fails** rather than skipping.
That is deliberate: `tst:4`/`tst:6`/`tst:11` exist so a missing sidecar cannot
pass vacuously, and `certificate_audit/fail_missing_stamp` pins it.

The consequence is a floor on which images the rule can meaningfully assess:

| image | outcome |
| --- | --- |
| built by apko **v1.2.30 or later** | assessable |

@0xDom-S 0xDom-S Sep 3, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖: The apko v1.2.30 sidecar-floor claim here (and the related version claims at :90, :187, and :407 for the ECS bundle bump v1.2.35→v1.2.43) are the only apko version references in the tree and nothing pins them in-repo. They rest entirely on the author's stated manual verification, which could not be checked offline in this review. Not a defect — flagged so reviewers know which claims carry no regression guard.

@stevebeattie stevebeattie Sep 10, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair flag, and it prompted the right fix. Rather than leave the claims resting on my say-so, all three are now re-derived from apko's pkg/build/certificates.go:

claim check
v1.2.30 first carries writeCABundleChecksums 0 occurrences at v1.2.29, 3 at v1.2.30
kaniko/ssl/certs/ca-certificates.crt not in caBundlePaths at v1.2.35, is at v1.2.43 absent at v1.2.35; at v1.2.43 the list gains "kaniko/ssl/certs/ca-certificates.crt", // Kaniko
ECS bundle still in caBundlePaths at v1.2.43 var/lib/ecs/deps/execute-command/certs/tls-ca-bundle.pem present

24981fc records the file and symbols in the doc so a future reader can re-check at any tag instead of trusting this PR. I deliberately did not add a test pinning apko's caBundlePaths: it would couple this repo's CI to a third party's source layout and break on an upstream refactor that changes nothing about the rule. Naming where to look seemed the better trade for a claim about a moving external dependency — open to the other view if you think the drift risk warrants it.

| built by apko **v1.2.29 or earlier** | fails — no sidecars exist |
| not built by apko at all | fails — no sidecars exist |

v1.2.30 is the first release carrying `writeCABundleChecksums`; v1.2.29 does not
have it. Every current Chainguard image is well past that, so this is not a
concern for scanning what the registry serves today. It matters when scanning
something older: an archived release, a customer's pinned image from before the
change, or an image built by other tooling.

Nothing in this repository pins any apko version, so the claims here and below
are about a moving external dependency. They live in apko's
`pkg/build/certificates.go` — `writeCABundleChecksums` and the `caBundlePaths`
list — and can be re-derived at any tag rather than taken on trust.

**A failure caused by the floor is not distinguishable from a real one by the
rule verdict alone** — both are `fail`. It *is* distinguishable from the scan
artifact, in the per-test OVAL results, so no access to the image is needed and
an archived results file can be read after the fact. Scan with `--oval-results`

@0xDom-S 0xDom-S Sep 3, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖: Two issues with the --oval-results guidance here.

(a) Invocation: this is the only occurrence of --oval-results/ARF anywhere in the repo; in OpenSCAP, --oval-results is functional only alongside --results. Worth spelling out the full invocation so a reader doesn't get an empty result — the repo's own harness (tests/oscap-offline/internal/scan/command.go:148-150, tests/e2e/run.sh:150-152) passes only --results and won't produce this per-test detail.

(b) Sensitivity: full OVAL results/ARF serialize textfilecontent54_item matched text for every rule in the profile, not just CertificateAudit — including /etc/shadow-matching rules co-selected in the same profile (e.g. UserPasswordConfiguredTest.xml, NoUsersCheck.xml use a regex whose [^:\n]* group captures the crypt field). This is bounded (check_existence="none_exist", instance=1, first match only, image already failing that rule) and not a vulnerability, but since :98-99 explicitly anticipates the file being read by someone with no image access, a clause noting full OVAL output is credential-bearing (or scoping the guidance to oscap oval eval --id oval:org.CABundleHash:def:1) is cheap and correct.

@stevebeattie stevebeattie Sep 10, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Split verdict on this one: (b) is right and is now in the doc; (a) is not.

(a) Invocation — not reproducible. --oval-results does not need --results. Tested both ways against the in-repo datastream:

# --oval-results alone
$ oscap xccdf eval --profile xccdf_basic_profile_.check \
                   --rule <CertificateAudit rule> --oval-results <ds>
CertificateAuditTest.xml.result.xml   DetectOpenSslTest.xml.result.xml
LibraryPermissionsTest.xml.result.xml NoUsersCheck.xml.result.xml
PackageSignatureTest.xml.result.xml   RemoteAccessServicesTest.xml.result.xml
UserPasswordConfiguredTest.xml.result.xml VarLogPermissionsTest.xml.result.xml

All eight per-test result files are produced; adding --results res.xml only adds res.xml alongside them. So the guidance was already correct and I have not changed it. What was genuinely missing is where the files land — CWD, one per OVAL document, named after it — so that is now stated, since a reader looking for a single named output would otherwise not find it.

Your point about the repo's own harness passing only --results (command.go:150, run.sh:152) is correct, but that is a property of the harness rather than of the flag.

(b) Sensitivity — correct, and demonstrated. Reproduced it rather than reasoning about it: against a synthetic root containing a planted canary,

$ grep -o '<ind-sys:text>[^<]*' oval-res.xml
<ind-sys:text>fakeuser:FAKEHASHCANARY123:

The matched line, password field included, is serialized verbatim.

One correction that strengthens it: this is not a hypothetical co-selection. 31 of the 91 rules xccdf_basic_profile_.check selects resolve to UserPasswordConfiguredTest.xml or NoUsersCheck.xml. The results also carry primary_host_name, os_version and architecture, so there is a host-identification dimension beyond the credential one.

The doc now flags a retained results file as credential-bearing and offers the scoped oscap oval eval --id oval:org.CABundleHash:def:1 alternative you suggested.

(or keep the ARF) and compare two tests:

| | no usable sidecar | trust store actually drifted |
| --- | --- | --- |
| `tst:4` — sidecar exists and parses | `false` | `true` |
| `tst:2` — bundle matches the sidecar digest | `error` | `false` |

`--oval-results` needs no companion flag and writes one file per OVAL document
into the working directory, named after it; `CertificateAuditTest.xml.result.xml`
is the one to read.

**Treat a retained OVAL results file as credential-bearing.** It carries fully
collected content for *every* definition in the profile, not only this one, and
`textfilecontent54` items serialize the matched line verbatim. This profile
co-selects the `/etc/shadow` rules, whose patterns match the password field, so
the results from an image failing one of those reproduce that field — along with
the scanned system's host name, OS version and architecture. Where the narrower
output will do, scope the run to this definition instead:

oscap oval eval --id oval:org.CABundleHash:def:1 \
--results cert-audit.xml <datastream>

The `error` on `tst:2` is itself the tell: the variable behind the comparison
collected no values, because there was no sidecar to read one from. A `false`
there means a sidecar was read and disagreed.

So `tst:4 false` says the rule *could not assess* this image — it predates the

@0xDom-S 0xDom-S Sep 3, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖: The disjunction "tst:4 false says the rule could not assess this image — it predates the mechanism, or the sidecar is malformed" omits post-build sidecar deletion.

Because tst:2's comparator comes from var:1←obj:4 and not from the bundle, the tst:4 false / tst:2 error signature is content-independent on the bundle side. A sidecar deleted after build — maliciously or via an image-slimming step — produces the identical signature to a pre-v1.2.30 image. The doc's own measurement at :117-118 was taken with the bundle intact, so it doesn't separate the two cases.

Note that in-image corroboration is not available either: sidecars are unowned (apk info -W → "Could not find owner package"), and apk audit always reports benign A for them, so the X ("owned but missing") code can never fire for a deleted sidecar (apk audit --system will NOT surface this).

Suggested fix: add "or the sidecar was removed after the build" to :111-112 and state that tst:4 false is not an exoneration. As a disambiguator, the bundle's actual SHA-256 is collected by obj:2 (filehash58_object, :100-103) and can be compared out-of-band against the signed build.

@stevebeattie stevebeattie Sep 10, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 24981fc.

Confirmed the content-independence: tst:2 uses ste:1, whose hash comes from var:1 ← obj:4, so the comparator is read from the sidecar and never from the bundle. Deleting the sidecar post-build therefore yields the same tst:4 false / tst:2 error pair as a pre-v1.2.30 image, and the doc's own measurement was indeed taken with the bundle intact, so it did not separate them.

Verified your in-image corroboration point rather than restating it — in wolfi-base:

$ apk info -W /etc/ssl/certs/.ca-certificates.crt.sha256
ERROR: ...: Could not find owner package
$ apk info -W /etc/ssl/certs/ca-certificates.crt
... is owned by ca-certificates-bundle-20260611-r0

So the sidecar is unowned while the bundle beside it is owned, which is exactly why apk audit can never report the sidecar as missing. That contrast is now in the doc, since it is what makes the "no in-image corroboration" claim concrete rather than assertive.

Also took the obj:2 disambiguator — it is an ind:filehash58_object over the bundle (:100-103), so the bundle's real SHA-256 is collected whatever tst:2 does, and comparing it out-of-band against a signed build is the thing that actually separates the two cases. Both that and "tst:4 false is not an exoneration" are now stated explicitly.

mechanism, the sidecar is malformed, or the sidecar was **removed after the
build**, whether by an image-slimming step or deliberately. That is a different
statement from "this image's trust stores were modified", but it is not an
exoneration either. Because `tst:2`'s comparator is read from the sidecar
(`var:1` ← `obj:4`) rather than from the bundle, a deleted sidecar produces the
same `tst:4 false` / `tst:2 error` signature as a pre-v1.2.30 image, and the
image itself will not settle which happened: the sidecars belong to no apk
package (`apk info -W` reports `Could not find owner package`, where the bundle
beside them is owned by `ca-certificates-bundle`), so `apk audit` cannot report
one as missing. What does help is `obj:2`, which collects the bundle's real
SHA-256 whatever `tst:2` does — compare it out-of-band against the digest from a
signed build.

`tst:4 true` with `tst:2 false` is the real finding.

The same reading applies to `tst:6`/`tst:7` for the Java truststore and
`tst:11`/`tst:12` for a `/kaniko` copy — **but only where that truststore or
copy is actually present.** On an image with neither, `tst:6`/`tst:11` are
`false` and `tst:7`/`tst:12` `error` as a matter of course, because their
objects collect nothing: that is the "no usable sidecar" column above appearing
on a fully assessable image, and on a *passing* one. `tst:5` and `tst:3` are
what carry those branches in that case, so read them first — an absent
truststore or `/kaniko` copy is not a finding.

Measured, not inferred: scanning an image with its sidecar removed and its
bundle intact gives `tst:4 false`, `tst:2 error`; scanning one with the sidecar
intact and the bundle appended to gives `tst:4 true`, `tst:2 false`. Both report
the rule as `fail`.

In-image, `ls -l /etc/ssl/certs/.ca-certificates.crt.sha256` answers the same
question more directly, where you have a shell and the image to hand.

Note this is a change in which images are assessable, not only in how. Under the
previous design the expected digest was pinned in the datastream, so an old image
could pass if its bundle happened to match that pin — no sidecar required.

## Why not a pinned digest

The rule previously pinned the CA bundle's SHA-256 in the datastream. That had
Expand Down Expand Up @@ -128,7 +221,20 @@ Then an `OR` for the Java truststore:
to the absent-truststore branch.
- **`tst:13` before falling back to `tst:9`.** A sidecar beside the `/kaniko`
copy takes precedence, so a divergent copy cannot sidestep its own sidecar by
appealing to the system one.
appealing to the system one. Which branch a real image takes has

@0xDom-S 0xDom-S Sep 3, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖: This new text (asserting the /kaniko copy now ships its own sidecar and real images have moved onto tst:11/tst:12) contradicts the unchanged deferral rationale later in the file at docs/certificate-audit.md:399-402, which still reads: "The copy is currently byte-identical to its system bundle, so the drift being guarded against is remote… Revisit if the /kaniko copy ever starts diverging, or gains a sidecar of its own." This PR's own evidence fires that revisit trigger verbatim, yet :193-194 concludes "nothing needs changing."

Two follow-ups worth doing in this PR (both landing in unchanged lines outside the diff, so not separately anchorable):

  • Update :399-402 so the file doesn't argue both sides.
  • The Known gap at :413-416 should note that the branch production images now take (obj:10) is the unguarded one — tests/stamps/run.sh:61 marks obj:10 required=no, and .github/workflows/update-ca-cert.yaml:35 stamps only cgr.dev/chainguard/jre:latest daily. Deferral may still be the right call; the rationale just needs restating on the new facts.

@stevebeattie stevebeattie Sep 10, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 24981fc, and you are right that the file was arguing both sides — the deferral names "gains a sidecar of its own" as its own revisit trigger, and the new text reports exactly that happening.

Verified both supporting facts:

  • tests/stamps/run.sh:61 — oval:org.CABundleHash:obj:10|kaniko/ssl/certs|...|no
  • .github/workflows/update-ca-cert.yaml — STAMP_IMAGE_REF: cgr.dev/chainguard/jre:latest, so the daily run stamps that image only

The deferral paragraph now states the original rationale, that it has been overtaken, and why the conclusion nevertheless stands — the two trust-surface additions are unchanged, and a divergent copy is now caught by tst:11/tst:12 against its own sidecar rather than needing the fallback. What the sidecar's arrival costs is guard coverage, not criteria coverage.

The Known gap is restated on the new facts and split in two, because the second half is the part that was invisible: the gap now falls on the branch production images actually take, since they resolve through obj:10 and obj:10 is the one sidecar the guard marks required=no. I kept required=no — older kaniko images legitimately ship none and the criteria fall back to tst:13/tst:9 for them — but the consequence is now written down: "present but wrong" is guarded there, "absent" is not.

run.sh's own comment had the same problem, arguing required=no from facts that have since changed, so it is corrected too (comment only).

changed. `kaniko/ssl/certs/ca-certificates.crt` was **not** in apko's
`caBundlePaths` at v1.2.35 but **is** at v1.2.43, and
`cgr.dev/chainguard-private/kaniko` now ships
`/kaniko/ssl/certs/.ca-certificates.crt.sha256` where in August 2026 it did
not. That private ref is the only kaniko image there is; no public
`chainguard/kaniko` exists to confuse it with. So real kaniko images have
moved off the fallback and onto `tst:11`/`tst:12`. Confirmed by running the guard
against it: the copy matches its own sidecar, and that sidecar matches
`obj:10`'s pattern — the first time that pattern has been checked against
anything other than a synthetic fixture. Both branches remain
fixture-covered, so the criteria need no change; the fallback is now the path
an *older* kaniko image would take. What this does change is which branch the
guards cover — see [Known gaps](#known-gaps).
- **`tst:5` uses `none_exist`** rather than testing for Java some other way,
because a non-Java image must not fail for lacking a truststore.

Expand Down Expand Up @@ -320,6 +426,16 @@ still fails with `No matching credentials were found for "cgr.dev"`. The
credential helper itself is usually already wired up (`chainctl auth
configure-docker` will say so); the audience is the part that goes missing.

To check the audience you actually need, ask for it by name:

chainctl auth status --audience cgr.dev

Without `--audience` the command reports on the console-API token, which is the
one that is valid whether or not the `cgr.dev` login has happened — so a bare
`chainctl auth status` is the reason this failure is easy to misdiagnose in both
directions. Note the flag is not a pure read: if the `cgr.dev` token has
expired, asking for its status re-requests one.

tests/stamps/run.sh cgr.dev/chainguard/jre:latest \
cgr.dev/chainguard-private/kaniko:latest

Expand All @@ -333,25 +449,49 @@ This is deliberately not automated. Doing so would need two additions to the
workflow's trust surface, not one: a credential for the private registry, and a
second accepted signer identity, because that image is signed by
`chainguard-dev/stereo/.github/workflows/release-containers.yaml` rather than
the `chainguard-images/images/*` identity the workflow requires. The copy is
currently byte-identical to its system bundle, so the drift being guarded
against is remote; the trade was judged not worth it for now. Revisit if the
`/kaniko` copy ever starts diverging, or gains a sidecar of its own.
the `chainguard-images/images/*` identity the workflow requires.

That trade was originally made on two grounds — the copy was byte-identical to
its system bundle, and it carried no sidecar of its own — and it named "gains a
sidecar of its own" as the thing that should prompt a revisit. Exactly one of
those has changed. Checked against the image directly:

| | |
|---|---|
| `kaniko/ssl/certs/.ca-certificates.crt.sha256` | now shipped (86 bytes) |
| `kaniko/ssl/certs/ca-certificates.crt` vs the system bundle | still byte-identical (`b8d83784…`) |
| the copy vs its own sidecar | matches |
| that sidecar vs `obj:10`'s pattern | matches, exactly one line |

So the revisit trigger has fired, but the divergence the guard exists to catch
still has not occurred.

The trade still looks right, for a narrower reason than before: the two
additions to the trust surface are unchanged, and the copy having its own
sidecar means a divergent copy would now be caught by `tst:11`/`tst:12` against
that sidecar rather than having to be caught by the fallback. What the sidecar's
arrival does cost is guard coverage rather than criteria coverage, which is
recorded under [Known gaps](#known-gaps).

## Known gaps

- apko also stamps `var/lib/ecs/deps/execute-command/certs/tls-ca-bundle.pem`
(still in `caBundlePaths` as of apko v1.2.35). This rule neither checks that
(still in `caBundlePaths` as of apko v1.2.43). This rule neither checks that
bundle nor its sidecar.
- `CertificateAuditTest.xml` is the only OVAL component referenced by the
datastream with no standalone file under
`gpos/xml/scap/ssg/content/ssg-chainguard-xccdf/OvalDefinitions/`, so
`make validate_checks` does not validate this definition.
- `<ind:instance>1</ind:instance>` with `only_one_exists` means a sidecar
containing two matching lines silently uses the first.
- Deleting `/etc/ssl/certs/java/cacerts` outright satisfies the Java `OR` via
`tst:5`.
- No automated run guards the `/kaniko` criteria against a real image; the
daily workflow inspects only public images. It is coverable on demand — see
daily workflow stamps only `cgr.dev/chainguard/jre:latest`, and the `/kaniko`
copy lives solely on a private ref. It is coverable on demand — see
[Covering the /kaniko criteria](#covering-the-kaniko-criteria) — and deferred
rather than declined.
- That gap now falls on the branch production images actually take. Since the
`/kaniko` copy gained its own sidecar, real images resolve through
`tst:11`/`tst:12` over `obj:10` — and `obj:10` is the one sidecar
`tests/stamps/run.sh` marks `required=no`, so a run that does reach a kaniko
image still will not fail on the sidecar going missing. `required=no` remains
correct, since older kaniko images legitimately ship no sidecar and the
criteria fall back to `tst:13`/`tst:9` for them; the consequence is that
"present but wrong" is guarded there while "absent" is not.
7 changes: 5 additions & 2 deletions tests/oscap-offline/internal/scan/fixtures_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -339,8 +339,11 @@ const caStampPath = "etc/ssl/certs/.ca-certificates.crt.sha256"
// caBundlePath and kanikoCABundlePath are the two locations CertificateAudit
// accepts SSL_CERT_FILE pointing at, as tar member paths (rootfs-relative, no
// leading slash). kanikoCAStampPath is the stamp file the /kaniko copy is
// checked against where one is shipped beside it; cgr.dev/chainguard/kaniko
// ships none today, so the copy falls back to the stamp at caStampPath.
// checked against where one is shipped beside it. The only kaniko image
// published is cgr.dev/chainguard-private/kaniko (there is no public one), and
// it does ship that sidecar as of apko v1.2.43, so real images now resolve
// through tst:11/tst:12; the fallback to caStampPath is what an older kaniko
// image takes. Both branches are covered by fixtures below.
const (
caBundlePath = "etc/ssl/certs/ca-certificates.crt"
kanikoCABundlePath = "kaniko/ssl/certs/ca-certificates.crt"
Expand Down
15 changes: 12 additions & 3 deletions tests/stamps/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,9 +52,18 @@ DEFAULT_IMAGES="${STAMP_IMAGES:-cgr.dev/chainguard/jre:latest}"
# sidecar (tst:13 + tst:9), so only a present-but-wrong sidecar
# is a failure.
#
# /kaniko is carried only by cgr.dev/chainguard-private/kaniko, so it is not
# reached by the default public image. Pass that ref explicitly to cover it
# wherever credentials for it exist; a run reports what it did not reach.
# obj:10 stays required=no because older kaniko images legitimately ship no
# sidecar beside the /kaniko copy and the criteria fall back for them. Note this
# is no longer the branch a current image takes: the copy entered apko's
# caBundlePaths by v1.2.43 and now carries its own sidecar, so production
# resolves through tst:11 + tst:12. The consequence of required=no is therefore
# that a sidecar going *missing* from an image that should have one will not
# fail this guard, only one that is present and wrong.
#
# /kaniko is carried only by cgr.dev/chainguard-private/kaniko — there is no
# public kaniko image — so it is not reached by the default public image. Pass
# that ref explicitly to cover it wherever credentials for it exist; a run
# reports what it did not reach.
SIDECARS=(
"oval:org.CABundleHash:obj:4|etc/ssl/certs|.ca-certificates.crt.sha256|ca-certificates.crt|yes"
"oval:org.CABundleHash:obj:6|etc/ssl/certs/java|.cacerts.sha256|cacerts|yes"
Expand Down