fix(ci): stop validate failing on stale generated_at date - #937
Merged
Conversation
The check-skill-catalog pre-commit hook (and validate.yml's required 'validate' check) compares scripts/generate_skill_index.py --check output byte-for-byte against the committed docs/skills/index.json and index.md. The catalog's generated_at field was always stamped with date.today() at build time, so the check failed whenever CI ran on a later calendar day than the last regeneration on main — regardless of whether any skill doc actually changed. This produced false-negative 'stale index' failures on 9 of 19 open PRs, none of which touched docs/skills content, training reviewers to ignore a required status check. Fix: only advance generated_at when the actual catalog content (schema_version + skills) differs from what's committed. --write becomes idempotent when nothing changed (no gratuitous date bump), and --check now tolerates a stale generated_at as long as the underlying skill data matches. A genuine drift (a skill doc edited without regenerating the index) still fails, since skills content itself differs — the real protection is preserved. Rejected alternatives: - Diffing against the PR merge result instead of PR head: doesn't fix the root cause, since the merge tree's freshly regenerated catalog would still get stamped with 'today' and be compared against a committed file stamped on a prior day. - Scoping the check to changed files: would blind it to a PR that edits a skill doc without regenerating the index, weakening the real protection this check exists for. - Dropping the committed generated file: much larger blast radius (index.md is referenced from docs), not needed once the byte-for-byte date comparison is fixed. - Auto-commit bot step: adds CI complexity for a problem solvable with a three-line comparison fix. Assisted-by: Claude Sonnet 5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
castrojo
pushed a commit
that referenced
this pull request
Aug 7, 2026
The check-skill-index pre-commit hook fails on main with: error: docs/SKILL.md is missing a link to skills/queue-feed.md Because pre-commit runs inside the required `validate` job, this failed CI for every open PR regardless of its content -- the same class of false-negative that #937 fixed for the generated skill index. queue-feed.md was added without a corresponding row in the SKILL.md routing table. Assisted-by: Claude Opus 5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The
validatecheck has been producing false-negative failures across the open PR backlog. At time of writing, 9 of 19 open PRs (#893, #917, #924, #926, #929, #931, #932, #933, #934) showed a redvalidatecheck unrelated to their own content:Root cause
scripts/generate_skill_index.pystampedgenerated_atwithdate.today()at script-execution time, then byte-compared the result against the committeddocs/skills/index.json.That means the check only passed if CI happened to run on the same calendar day the index was last regenerated on
main— entirely independent of whether any skill doc actually changed. Any PR whose CI ran a day or more after the last regeneration failed.Reproduced locally with zero content edits:
--checkfailed purely because days had passed since the last commit toindex.json.This matters because
validateis a required status check in the merge queue ruleset, so the noise trains reviewers to ignore a gate that is supposed to block merges.Fix
Only advance
generated_atwhen the catalog's actual content (schema_version+skills) differs from what is committed.--writebecomes idempotent when nothing changed;--checktolerates a stale date as long as the skill data matches.Verification
Proved in both directions, with a regression test for each:
2020-01-01with zero skill content changes;--checknow passes (previously failed). Covered bytest_generate_skill_index_check_tolerates_stale_generated_at.--checkstill fails withindex.json is stale. Covered bytest_generate_skill_index_check_still_fails_on_real_drift.just checkpasses. Skill doc test suite: 8 passed.Alternatives rejected
Assisted-by: Claude Sonnet 5 via GitHub Copilot