Skip to content

Make the datastream's OVAL reference sources say Custom, and gate it - #171

Merged
stevebeattie merged 1 commit into
chainguard-dev:mainfrom
stevebeattie:cis-to-custom
Sep 1, 2026
Merged

stevebeattie merged 1 commit into
chainguard-dev:mainfrom
stevebeattie:cis-to-custom

Conversation

@stevebeattie

Copy link
Copy Markdown
Member

Problem

Five OVAL definitions embedded in the datastream carried
<reference source="CIS"/> while every standalone file said Custom — and
two embedded blocks said Custom too. So the datastream disagreed both with its
sources and with itself.

Custom is the intended value — this is a missed hand-sync, not a judgement call

Commit 3f69d3f ("All of the formatting", 2025-07-18) deliberately changed
exactly these five references from CIS to Custom in the standalone files. It
touched the datastream in the same commit — 3402 lines — and left its
reference sources alone. The standalone value has been Custom ever since.

Supporting evidence: the elements carry no ref_id, and these are
Chainguard-authored checks for a DISA GPOS SRG profile rather than CIS benchmark
content, so CIS asserted a provenance that was not real.

This is the second confirmed instance of the same failure class as
openssh-sftp-server in #162: a change applied to the standalone files and not
to the datastream.

Is it user-visible?

Partly, and worth being precise about — I tested rather than assumed:

Surface CIS present?
oscap HTML report no (0 occurrences)
machine-readable results XML from a scan yes
the published datastream itself yes

No scan verdict changes. But it reaches the compliance evidence a customer
retains, and the shipped content.

Change

Convert the five, and close the gate behind them: descriptive differences in
the mirror check now fail rather than being logged.

That exemption existed only because these five instances did — failing on a
difference that changes no verdict would have meant disabling the check instead
of fixing content. With the content fixed, the exemption goes, so this class
cannot drift back in silently.

Verification

  • No CIS values remain anywhere in the repo; datastream and standalone files
    now both report Custom exclusively.
  • Mirror check: zero descriptive differences across all 8 files.
  • Reintroducing a single CIS now fails the check, naming the file and
    definition — confirmed by mutation, then reverted.
  • gofmt/go vet clean, validate_checks passes, sds-validate rc=0, and
    oscap xccdf validate reports the same 235 pre-existing errors as main.

🤖 Generated with Claude Code

Five OVAL definitions embedded in the datastream carried
<reference source="CIS"/> while every standalone file said "Custom", and two
embedded blocks said "Custom" as well — so the datastream disagreed both with
its sources and with itself.

"Custom" is the intended value, and this is a missed hand-sync rather than a
judgement call. Commit 3f69d3f ("All of the formatting", 2025-07-18)
deliberately changed exactly these five references from CIS to Custom in the
standalone files. It touched the datastream in the same commit — 3402 lines —
but left its reference sources alone. The value has never been anything but
Custom in the standalone files since. These are Chainguard-authored checks for
a DISA GPOS SRG profile, not CIS benchmark content, and the elements carry no
ref_id, so "CIS" asserted a provenance that was not real.

This does not change any scan verdict, but it is not invisible either. The
value is absent from the HTML report oscap renders, and present in the
machine-readable results a scan emits — so it reaches compliance evidence
customers retain, as well as the published datastream itself.

Convert the five, and close the gate behind them: descriptive differences in
the mirror check now fail rather than being logged. They were logged
deliberately while these five existed, because failing on a difference that
changes no verdict would have meant disabling the check instead of fixing
content. The content is fixed, so the exemption goes.

Verified: no CIS values remain anywhere; the mirror check reports zero
descriptive differences across all 8 files; reintroducing a single CIS now
fails the check, naming the file and definition. gofmt/vet clean,
validate_checks passes, sds-validate rc=0, and oscap xccdf validate reports
the same 235 pre-existing errors as main.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@stevebeattie
stevebeattie requested a review from egibs August 31, 2026 22:32
@stevebeattie
stevebeattie merged commit ffdec6a into chainguard-dev:main Sep 1, 2026
3 checks passed
@stevebeattie
stevebeattie deleted the cis-to-custom branch September 1, 2026 07:34
stevebeattie added a commit that referenced this pull request Sep 24, 2026
The mirror check reported three classes of difference and failed on two of
them. `report.Unmirrored` -- definitions embedded in the datastream with no
standalone file -- was only logged, because the stub OVAL definition was its
one permanent entry and failing while that stub was still embedded would have
meant disabling the check rather than fixing content.

#172 removed the stub, so the list is empty and the gate can close behind it.
Verified both directions rather than just that the suite stays green:

  - passes on current content (8 standalone files compared, list empty), so
    this is a no-op today;
  - moving DetectOpenSslTest.xml out of the standalone directory makes its
    embedded block unmirrored and the test fails naming
    `oval:org.OpenSsl:def:1`, with the remedy in the message.

What this guards is narrow but real: oscap-playground's `has_stub_oval`
metadata flag can reintroduce a datastream-only component on a regeneration,
and nothing else in the suite would notice. The standalone-file-without-a-block
direction was already covered; this is the other one.

Also corrects the function's doc comment, which still claimed descriptive
differences were logged. They have failed since #171 closed that gate, so the
comment described neither the body nor the behaviour. It now names all three
classes and records that "logged" is the state to return to if a future
disagreement is judged not worth fixing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants