Skip to content

feat: gate stable promotion on security qualification instead of pre-release publication #3559

Open
0 of 1 issue completed
Open
feat: gate stable promotion on security qualification instead of pre-release publication#3559
0 of 1 issue completed
Assignees
purp

Description

@purp
purp
opened on Sep 22, 2026

User Story

As a maintainer running the weekly release train, I want security scan findings
to gate promotion to stable rather than building and publishing a
pre-release, so that a static-analysis backlog cannot stop the release
pipeline from producing artifacts to qualify.

Problem Statement

release-tag.yml calls the aggregate Security Scan as a precondition for
publication. Any blocking finding from Codex Security, CodeQL, Trivy, Zizmor, or
Cargo Deny fails the workflow before artifacts are published.

RFC 0014 specifies the opposite ordering. In
rfc/0014-release-stability/README.md (Pre-release qualification and stable
publication), every pre-release is tagged and built first, and the four
qualification suites — including Security review, which "verifies security
scan results, reviews changes to security-sensitive boundaries, and confirms
that every finding has the required disposition" — run against the published
artifacts. Qualification determines eligibility for stable promotion; it is not
a precondition for the pre-release existing.

The current wiring conflates the two. A finding backlog unrelated to the
candidate commit stops artifact production entirely.

Impact / Why This Matters

Observed on release-tag run 35725722342 (2026-09-22): the nightly failed with
8 HIGH/CRITICAL CodeQL findings, 27 HIGH Zizmor findings, 1 HIGH Trivy finding,
and one Cargo Deny advisory (RUSTSEC-2026-0295). No pre-release artifacts were
produced, so nothing was available to qualify, and the release train stalled.

The current workaround is the temporary fail-on-static-findings: false
override added in #3552, which makes CodeQL, Trivy, and Zizmor informational on
the release path. That unblocks the train but removes the signal entirely rather
than relocating it — there is no gate anywhere that stops a stable promotion on
those findings, and the override is documented only in prose in CI.md and
architecture/build.md.

The workaround is insufficient because it trades a false blocker for no blocker.
RFC 0014's model keeps the signal and puts it where it belongs.

Proposed Design

User-facing workflow

  1. A pre-release tag always builds and publishes its artifacts. Scanner
    execution errors still fail the build, because a scan that did not run
    produces no evidence.
  2. Security scans run as a qualification suite against the published
    pre-release, alongside conformance, upgrade, and breaking-API review.
  3. The suite's result — per-scanner counts and the disposition of each finding —
    is recorded in the pre-release manifest described in RFC 0014.
  4. Stable promotion requires the security suite to pass. A pre-release with
    undisposed blocking findings is ineligible for promotion and is reported as
    such, rather than silently skipped.
  5. Maintainers dispose of findings (fix, accept with justification, or mark
    false positive) against a pre-release, and that disposition is what the
    promotion gate reads.

Externally observable behavior

  • A finding backlog delays stable promotion; it never prevents a pre-release
    from being built or published.
  • The reason a pre-release was not promoted is visible in its manifest and
    qualification result, not only in a failed workflow log.
  • fail-on-codex-findings and fail-on-static-findings on security-scan.yml
    are no longer needed on the release path once the qualification gate exists,
    and the temporary override in release-tag.yml is removed.

Internal implementation — where the suite runs, how dispositions are stored, and
the manifest schema — is deliberately left open.

Acceptance Criteria

  • Pre-release tags build and publish artifacts regardless of scanner
    findings; only scanner execution failures block the build.
  • Security scan results are attached to the pre-release manifest with
    per-scanner finding counts.
  • Stable promotion is blocked when the security qualification suite has
    undisposed blocking findings, and the block is reported with the specific
    findings responsible.
  • The temporary fail-on-static-findings: false override is removed from
    release-tag.yml.
  • CI.md, architecture/build.md, and the RFC 0014 release qualification
    supplement describe the enforcement point consistently.
  • A pre-release with a known backlog is demonstrated publishing successfully
    and being correctly withheld from stable promotion.

Alternatives Considered

  • Baseline/delta enforcement inside release-tag.yml. Keep publication
    gated, but only fail on findings new since the previous stable tag. This is
    the removal condition currently documented for the ci(security): stage release finding enforcement #3552 override. It fixes
    the immediate backlog problem but keeps enforcement in the publication path,
    so it still contradicts RFC 0014 and still lets a single new finding block
    artifact production.
  • Keep blocking and burn down the backlog. Triage the 8 CodeQL, 27 Zizmor,
    and 1 Trivy findings to zero, then restore fail-on-static-findings: true.
    This does not change the structural problem: the next backlog stalls the train
    again, and it couples release cadence to triage capacity.
  • Leave the override in place indefinitely. Rejected — it removes the signal
    with no compensating gate before stable promotion.

Agent Investigation

Related material:

Activity

  1. purp
    self-assigned this
    on Sep 22, 2026

  2. purp commented on Sep 23, 2026

    @purp
    purp
    on Sep 23, 2026
    CollaboratorAuthor

    RFC 0014 Release Qualification: Design

    Status: draft for review. Tracks #3559. Related: #3632 (promotion vs. rebuild).

    Goal

    Move release enforcement out of pre-release publication and into a
    qualification stage that gates stable promotion, as RFC 0014 specifies. Build
    the substrate (artifact manifest, durable qualification results, promotion
    gate) with security and conformance as the first implemented suites. Upgrade
    and breaking-API review are declared but not yet implemented.

    Scope

    In scope: the OpenShell GitHub repository's release workflows, the pre-release
    artifact store, qualification results, security finding dispositions, and
    stable promotion.

    Out of scope:

    • Storage cost.
    • Pipelines outside the OpenShell GitHub project, including what happens to the
      Python wheel after it's built.
    • How other people and systems read GHCR artifacts.
    • The TypeScript SDK. If it returns to scope, it's packaged with the release.
    • Standalone fixes to the current process (for example, attesting snaps). Those
      are peeled off as individual fixes once validated and agreed by maintainers.

    Current state

    PR #3593 (merged 2026-09-23) is the first increment of this design:

    • A Release Qualification job in release-tag.yml aggregates security,
      conformance integration, feature integration, Docker E2E, and VM E2E results
      into the release-tag-v1 profile.
    • Intended: pre-release artifact assembly and publication no longer depend on
      that profile passing. Stable publication still does, in the same run.
      Observed: this doesn't work yet. In the v0.1.0-pre.10 run
      (35888840979), Release Qualification and every direct needs of Release
      succeeded, but Release was skipped, along with image tagging, Helm, and the
      qualification summary publish. GitHub skips a job with the default success()
      condition when any ancestor job failed, and the suite jobs are ancestors
      through qualification-result. Release and its downstream jobs need an
      explicit condition such as !cancelled() plus success of their direct needs.
    • Each attempt writes qualification-summary.json (schema v1) to the run
      summary, a 90-day Actions artifact, and
      ghcr.io/nvidia/openshell/qualification:<version>-run-<run-id>-attempt-<attempt>.
    • The summary records rfc_0014_complete: false and lists the missing suites:
      upgrade, breaking API change review, and the remaining conformance
      configurations.
    • Strict High/Critical reporting is restored for all scanners
      (fail-on-static-findings: true).

    Consequence: with strict reporting restored, any undispositioned High/Critical
    finding fails the profile and blocks stable. In v0.1.0-pre.10, Trivy, Codex
    Security, Zizmor High, and CodeQL (rust) failed, as did Docker E2E (python).

    Still missing relative to RFC 0014: an artifact manifest and durable artifact
    store, per-suite results, finding dispositions, promotion of the newest
    eligible pre-release, qualification decoupled from the build, and the upgrade
    and breaking-API suites.

    Section 1: Artifact storage and manifest

    Each pre-release pushes one immutable OCI artifact,
    ghcr.io/nvidia/openshell/release-assets:<version>, holding the file set a
    stable GitHub Release would carry (tarballs, deb, rpm, snap, wheel, checksum
    files, Homebrew formula) plus manifest.json.

    The manifest records:

    • Identity: version, source commit, producing workflow run.
    • Build inputs: hashes of Cargo.lock, flake.lock, rust-toolchain.toml,
      mise.lock, and the vm-runtime hash.
    • Artifact digests: every file, plus gateway, sandbox, and supervisor image
      digests.
    • SBOMs: per-binary CycloneDX from Syft (reading cargo auditable metadata),
      and references to the image SBOM attestations.
    • Attestations: references to the actions/attest output.

    Pre-releases stay off the curated GitHub Releases page; the OCI store is their
    index, following the Bazel rolling-release intent. Fallback, if GHCR storage
    proves unworkable: the existing 90-day Actions artifact bundles, which
    install.sh's OPENSHELL_VERSION=pre path already consumes. (A GitHub Release
    with prerelease: true was the earlier fallback; it's dropped because it puts
    pre-releases on the curated Releases page.)

    Unchanged: the five 90-day pre-release Actions bundles and the install.sh
    pre path that reads them.

    Rebuild vs. promotion

    Stable rebuilds the qualified commit rather than promoting the qualified bytes.
    This is undesirable: each rebuild is another opportunity for a supply-chain
    compromise to enter artifacts. We accept it for now in the name of progress.
    Every artifact embeds its version (binaries, default supervisor image tag,
    images, deb, rpm, wheel, snap, Helm chart), and RFC 0014 keeps versions out of
    source, so the stable version can't exist in any artifact until after
    qualification. #3632 decides promotion vs. rebuild. If it chooses promotion,
    that's a separate project.

    Section 2: Promotion gate

    Suite declarations

    A repository config file defines the qualification profile: every suite, and
    every conformance configuration, is either implemented or declared
    not-implemented. It replaces the hard-coded missing_suites list in
    generate-qualification-summary.sh and versions the profile name
    (release-tag-v1, -v2, ...).

    • Implemented: must pass. fail, error, or a missing result blocks.
    • Declared not-implemented: records not-implemented; advisory only. It appears
      in the promotion summary and never blocks.
    • An implemented suite reporting not-implemented is treated as error. This
      follows the RFC rule that undeclared skips fail qualification.

    Initial profile: security, conformance integration, feature integration, Docker
    E2E, and VM E2E implemented (the #3593 set); upgrade, breaking-API review, and
    the remaining conformance configurations declared not-implemented.

    release-promote.yml

    Triggers: Tuesday schedule, and manual dispatch with an optional pre-release
    version and a dry-run flag. Manual dispatch covers out-of-band patch or minor
    releases on any day (for example, an urgent security fix); it still requires
    eligibility.

    Train: the newest vX.Y.Z-pre.N base with no stable tag. Choosing a patch or
    minor train stays a manual maintainer decision; a started train never switches.

    Nothing new to release: if no -pre.N candidate has commits beyond the latest
    stable tag (for example, an out-of-band release already shipped this train's
    newest candidate), the run skips. It succeeds with a summary saying so, and
    publishes nothing. This keeps the weekly cron from re-releasing an out-of-band
    release.

    Selection: walk -pre.N newest to oldest; a candidate is eligible when the
    latest result for every suite implemented in the current profile is pass.
    Promote the first eligible candidate. If none qualifies, no release: the run
    fails, and its summary lists each candidate and its blocking suites.

    Suites added since a candidate was qualified: each result records the profile
    version it ran under. When the current profile implements a suite that a
    candidate has no result for, promotion runs that suite against the candidate's
    stored artifacts via release-qualify.yml before deciding eligibility, rather
    than failing on the missing result. Under today's rebuild model the final
    checks also cover this; under promotion (#3632) this is the only point where
    new critical tests reach the artifact before it ships.

    Promotion:

    1. Rebuild at the manifest's source commit with the stable version, reusing
      release-tag.yml build jobs via workflow_call, after checking build inputs
      against the manifest.
    2. Final checks: the rebuild is a new artifact, so it gets the full implemented
      profile, as release-tag.yml does for stable today, plus --version and SBOM
      checks. This qualifies the bytes that actually ship, which partly offsets the
      rebuild risk in Section 1. If chore(release): evaluate promoting qualified pre-release artifacts instead of rebuilding for stable #3632 moves to promotion, prior results on the
      promoted artifacts stand, and only suites added since they were qualified run
      (see Selection).
    3. Create vX.Y.Z on the source commit and run the existing publish jobs
      (GitHub Release, image retag, Helm, snap, docs).
    4. Delete that commit's -pre.N tags.

    Pushing a stable tag by hand no longer publishes; release-promote.yml creates
    the tag as its last step. Maintenance branches are out of scope.

    Section 3: Running qualification

    Pre-release publication in release-tag.yml stops depending on the suites (intended by
    #3593; see Current state for the remaining skip).

    release-qualify.yml runs qualification separately from the build:

    • Triggered on completion of a pre-release release-tag.yml run, or by manual
      dispatch with a version and optional suite list.
    • Pulls release-assets:<version> by digest and verifies every file against
      the manifest before any suite runs. Suites use stored artifacts and images by
      digest, never the build run's workflow artifacts.
    • One job per implemented suite; the existing security and conformance/E2E jobs
      move here.
    • One job writes not-implemented results for declared suites, so every
      pre-release has a result for every suite.

    Results extend #3593's format rather than replacing it: standalone OCI
    artifacts in ghcr.io/nvidia/openshell/qualification, one per suite per
    attempt, tagged <version>-<suite>-run-<run-id>-attempt-<attempt>, annotated
    with the source revision, version, and release-assets manifest digest. This
    avoids GHCR's missing OCI referrers API (verified: 404 on the referrers
    endpoint). Schema v2 records suite, profile version, status (pass, fail, not-implemented,
    error), run URL, timestamp, and details. Latest result per suite wins.
    Results are append-only.

    Errors and reruns: infrastructure failures record error. A manual dispatch
    for one suite re-runs it; the new result supersedes the old. Code fixes produce
    a new pre-release; reruns are for flaky or broken infrastructure.

    What separating qualification from the build enables

    • Re-evaluating security after a disposition merges, without a new pre-release.
    • Re-running a single suite after an infrastructure error.
    • Trying a new suite (for example, upgrade tests) against existing pre-releases
      before declaring it implemented.
    • Re-qualifying an older pre-release when the newest fails, so Tuesday still
      has a candidate.

    Section 4: Security finding dispositions

    RFC 0014: security review passes "when no unresolved critical or high-severity
    finding affects a supported configuration."

    security-dispositions.yaml is the unified record of risk decisions. Each
    entry: finding ID, disposition, reason, approver, and (for accepted risk) an
    expiry date.

    • false-positive: never blocks.
    • not-affected: doesn't touch a supported configuration; never blocks.
    • accepted-risk: non-blocking until expiry; always listed in results and the
      promotion summary.

    Tool-native files remain because the tools read them. Every accepted risk in a
    native file must also appear in security-dispositions.yaml; the security suite
    treats a native suppression without a matching, unexpired entry as
    undispositioned. Today that means the four deny.toml ignores. The
    .trivyignore.yaml file holds only false positives by its own rule and stays
    as is.

    Dispositions are read from main at qualification time, not from the
    pre-release's source commit, and the result records the dispositions commit.
    Pass when every High/Critical finding has a current disposition; otherwise
    fail, listing undispositioned findings. Changes go through PRs and CODEOWNERS.

    Rejected: GitHub code-scanning dismissals (SARIF-only, not code-reviewed).

    Section 5: Incremental delivery

    Each increment lands on its own. Before starting the next, confirm: main CI
    stays green, a dev release succeeds, and a manually dispatched pre-release
    publishes its complete artifact set and records qualification. Increments that
    touch the stable path also need a stable dry run.

    1. Landed: ci(release): publish prereleases with qualification summary #3593 (+ fix(ci): use approved setup-oras revision #3625 ORAS pin fix). In-workflow release-tag-v1
      profile, summary v1 in GHCR, stable gated.
      • Follow-up fix: the publication skip. Give Release, Tag GHCR Images, Release Helm Chart (OCI), and Publish Qualification Summary (OCI) explicit conditions so a failed suite no longer skips pre-release
        publication. Verify with a pre-release whose profile fails.
    2. Security dispositions. security-dispositions.yaml, backfill the
      deny.toml accepted risks, native-file consistency check, and a security
      result that honors dispositions. First because strict reporting now blocks
      stable on the existing finding backlog.
    3. Profile config and summary v2. Move implemented/not-implemented
      declarations out of the script; per-suite result documents; advisory
      not-implemented. Still produced inside release-tag.yml.
    4. Manifest and release-assets artifact. Additive push from the pre-release
      path; summaries gain the manifest digest annotation.
    5. release-qualify.yml in shadow. Runs against stored artifacts alongside
      the in-workflow profile; compare results across several pre-releases.
    6. Cut over qualification. Remove suite jobs from the pre-release path of
      release-tag.yml; release-qualify.yml becomes the source of results.
    7. release-promote.yml dry run. Tuesday selection only; compare with
      maintainer judgment.
    8. Cut over promotion. Remove the stable tag-push trigger; promotion
      rebuilds, re-runs the profile, tags, publishes, and deletes -pre.N tags.

    Independent follow-ups, each validated and agreed before landing:

    • Scheduled security scans for early signal between releases (daily CodeQL and
      Cargo Deny and weekly workflow-security crons exist; Trivy changes run on PRs;
      Codex Security and the image Trivy scan run only when called from a release
      or dispatched by hand).
    • Nightly CI gate ("last green main"): deferred; main CI isn't stable enough
      yet. Nightly remains main HEAD at tag time.
    • Breaking-API suite (buf breaking is configured in buf.yaml but never run).
    • Upgrade suite (U01–U06).
    • Attesting snaps.
    • chore(release): evaluate promoting qualified pre-release artifacts instead of rebuilding for stable #3632: promotion vs. rebuild.

    Testing

    • Eligibility selection, profile evaluation, and the disposition consistency
      check live in scripts under tasks/scripts/ with unit tests, following
      test-qualification-summary.sh and sync_docs_website_test.py.
    • Workflows are exercised by manual dispatch against existing pre-releases and
      through release-promote.yml dry runs.
    • Existing actionlint and Zizmor checks cover workflow syntax and security.

    Decisions recorded

    • Cadence after an out-of-band release: the Tuesday cron still runs. It skips
      when there's nothing newer than the latest stable release (Section 2).
    • Stable final checks run the full implemented profile against the rebuilt
      artifacts.
    • Fallback storage is the existing 90-day Actions artifact bundles.
  3. purp
    added a sub-issue on Sep 24, 2026

  4. github-actions commented on Oct 8, 2026

    @github-actions
    github-actionsBot
    on Oct 8, 2026 – with GitHub Actions

    This issue has had no activity for 14 days and is now marked stale. It may be closed in 7 days if there is no further activity. Comment or remove the state:stale label to keep it open.

  5. github-actions
    added
    state:staleInactive item at risk of automatic closure.
    on Oct 8, 2026
  6. purp
    removed
    state:staleInactive item at risk of automatic closure.
    on Oct 8, 2026

  7. elezar commented on Oct 12, 2026

    @elezar
    elezar
    on Oct 12, 2026
    Member

    Proposed follow-up for the stable publication gate: use a GitHub Environment with required reviewers to provide an approval path when qualification fails. Release Ready could require successful artifact builds plus either successful qualification or a successful environment-protected approval job. The approval job would run only for stable candidates that fail qualification; prereleases would continue publishing automatically despite qualification failures.

    Approval should be recorded as an explicit exception, preserving the failed qualification evidence. For now, the local workflow change keeps stable qualification mandatory and only adds a placeholder for this future approval path.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions