Skip to content

audit: warn when action_shas present and --resolve is off #57

Description

UX: warn when action_shas is present and --resolve is off

Problem

nboot audit runs offline by default — --resolve is off, so action
SHAs are left as placeholder values in the rendered plan. This is
intentional and documented (audit.md "By default audit runs
offline …"; the CLI --help says the same), so this isn't a hidden
bug.

The improvement opportunity: when a pack declares action_shas AND
the user runs offline, compute_diffs will report every affected
workflow as drifted because the rendered placeholders don't match
the on-disk real SHAs. The user has no on-screen cue that the
finding count would collapse if they passed --resolve. They then
either (a) treat the diff as a real drift list and waste triage
effort, or (b) eventually re-read the docs and re-run.

This is a UX gap, not a defect — the docs already cover it, but the
runtime experience doesn't reinforce it at the moment of confusion.

Proposed resolution

Single approach: smart warning at audit time.

run_audit() already has both skip_resolve and the loaded manifest
in scope at audit.py:133-136. When manifest.get("action_shas") is
non-empty and skip_resolve=True, emit a one-line note to stderr
before running compute_diffs:

Notice: --resolve is off and pack '<name>' declares N action SHA(s).
Workflow files rendered with placeholder SHAs may show as drift even
when the target is fully conforming. Re-run with --resolve (requires
gh) to compare against real SHAs.

Don't fail; don't suppress findings; let the user decide.

Pair with a one-paragraph addition to audit.md "Operational notes"
linking the flag and the manifest field.

Why "warning, not docs-only"

The docs path was suggested but rejected — the existing docs already
state the offline default clearly. Repeating the warning in CLI help
won't reach users who don't pass --help. A runtime warning fires
exactly when the user is about to be confused.

A. Smart warning (preferred)

When the rendered plan contains action-SHA placeholders AND
--resolve is off, emit a one-line warning to stderr:

Notice: --resolve off; N action-SHA placeholders in the rendered
plan will report as drift if the target was rendered online.
Re-run with --resolve to compare against real SHAs (requires gh).

Don't fail; let the user decide. Pair with a doc paragraph in
docs/reference/audit.md's "Operational notes" section.

B. Documentation only

Keep current behaviour, add a clear caveat in the --help text and
docs explaining that audits against pack-rendered repos generally
need --resolve to be meaningful.

Acceptance criteria

  • When manifest["action_shas"] is non-empty and skip_resolve
    is True, run_audit emits a one-line note to stderr naming
    the count and the recommended remediation (--resolve).
  • When --resolve is passed (online), no notice is emitted.
  • When the pack declares no action_shas, no notice is emitted
    regardless of --resolve.
  • docs/reference/audit.md "Operational notes" gains a
    paragraph linking the manifest field and the flag, with a
    copy-pasteable --resolve example.
  • Test in tests/test_audit.py asserts the notice is emitted
    (capsys / caplog) only in the warning-eligible case.

Effort

~30 minutes including tests + doc paragraph.

Source

Bot review on PR #51 (Grippy audit.py:175 MEDIUM
"Network-Skipping Mode May Cause Audits to Miss SHA Drift"),
sharpened post-Codex review (the original "silent footgun" framing
was wrong — docs do cover this — so this is a UX-warning
improvement, not a bug fix).

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions