Skip to content

fix(ci): stop validate failing on stale generated_at date - #937

Merged
castrojo merged 1 commit into
mainfrom
fix/validate-skill-index-drift
Aug 7, 2026
Merged

fix(ci): stop validate failing on stale generated_at date#937
castrojo merged 1 commit into
mainfrom
fix/validate-skill-index-drift

Conversation

@castrojo

@castrojo castrojo commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Problem

The validate check has been producing false-negative failures across the open PR backlog. At time of writing, 9 of 19 open PRs (#893, #917, #924, #926, #929, #931, #932, #933, #934) showed a red validate check unrelated to their own content:

error: docs/skills/index.json is stale. Run scripts/generate_skill_index.py --write

Root cause

scripts/generate_skill_index.py stamped generated_at with date.today() at script-execution time, then byte-compared the result against the committed docs/skills/index.json.

That means the check only passed if CI happened to run on the same calendar day the index was last regenerated on main — entirely independent of whether any skill doc actually changed. Any PR whose CI ran a day or more after the last regeneration failed.

Reproduced locally with zero content edits: --check failed purely because days had passed since the last commit to index.json.

This matters because validate is a required status check in the merge queue ruleset, so the noise trains reviewers to ignore a gate that is supposed to block merges.

Fix

Only advance generated_at when the catalog's actual content (schema_version + skills) differs from what is committed. --write becomes idempotent when nothing changed; --check tolerates a stale date as long as the skill data matches.

Verification

Proved in both directions, with a regression test for each:

  • No false negative — backdated the committed index to 2020-01-01 with zero skill content changes; --check now passes (previously failed). Covered by test_generate_skill_index_check_tolerates_stale_generated_at.
  • Real drift still caught — edited a skill's front matter without regenerating; --check still fails with index.json is stale. Covered by test_generate_skill_index_check_still_fails_on_real_drift.

just check passes. Skill doc test suite: 8 passed.

Alternatives rejected

  • Compare against the merge result instead of PR head — doesn't address the cause; the merge tree still gets stamped "today".
  • Scope the check to changed files — would blind it to a PR that edits a skill doc without regenerating the index.
  • Stop committing generated files — far larger blast radius for a date-comparison bug.
  • Auto-commit bot step in CI — added complexity over a small fix.

Assisted-by: Claude Sonnet 5 via GitHub Copilot

The check-skill-catalog pre-commit hook (and validate.yml's required
'validate' check) compares scripts/generate_skill_index.py --check output
byte-for-byte against the committed docs/skills/index.json and index.md.
The catalog's generated_at field was always stamped with date.today() at
build time, so the check failed whenever CI ran on a later calendar day
than the last regeneration on main — regardless of whether any skill doc
actually changed. This produced false-negative 'stale index' failures on
9 of 19 open PRs, none of which touched docs/skills content, training
reviewers to ignore a required status check.

Fix: only advance generated_at when the actual catalog content
(schema_version + skills) differs from what's committed. --write becomes
idempotent when nothing changed (no gratuitous date bump), and --check
now tolerates a stale generated_at as long as the underlying skill data
matches. A genuine drift (a skill doc edited without regenerating the
index) still fails, since skills content itself differs — the real
protection is preserved.

Rejected alternatives:
- Diffing against the PR merge result instead of PR head: doesn't fix
  the root cause, since the merge tree's freshly regenerated catalog
  would still get stamped with 'today' and be compared against a
  committed file stamped on a prior day.
- Scoping the check to changed files: would blind it to a PR that edits
  a skill doc without regenerating the index, weakening the real
  protection this check exists for.
- Dropping the committed generated file: much larger blast radius
  (index.md is referenced from docs), not needed once the byte-for-byte
  date comparison is fixed.
- Auto-commit bot step: adds CI complexity for a problem solvable with a
  three-line comparison fix.

Assisted-by: Claude Sonnet 5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@castrojo
castrojo merged commit 71c28e5 into main Aug 7, 2026
4 of 5 checks passed
@castrojo
castrojo deleted the fix/validate-skill-index-drift branch August 7, 2026 00:41
castrojo pushed a commit that referenced this pull request Aug 7, 2026
The check-skill-index pre-commit hook fails on main with:

  error: docs/SKILL.md is missing a link to skills/queue-feed.md

Because pre-commit runs inside the required `validate` job, this failed
CI for every open PR regardless of its content -- the same class of
false-negative that #937 fixed for the generated skill index.

queue-feed.md was added without a corresponding row in the SKILL.md
routing table.

Assisted-by: Claude Opus 5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant