Repository navigation
685: Turn every incident into a check and add alert triage to codebase-architecture - #61
Conversation
Incidents and repeated mistakes now end in a deterministic check, lint config, allowlists and new env vars escalate to a human, judgement gates start in shadow, and agent-runtime gains an Alert triage section whose Alert is wrong draft is the one exception to the fix-job test and protected-path rules. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QS4mU5DxP81inR3AhxCcKy
Deleting or muting an alert is never Alert is wrong, triage runs under the fix job's isolation rules, the issue carries the fingerprint, and rung 3 now matches the escalation list. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QS4mU5DxP81inR3AhxCcKy
| Verdict: pass. 1 feature covered, 1 skipped (behavioural with/without-skill runs, which need separate isolated sessions), 0 failed. Every Done-when item holds at 863d4d2. The simplify pass in 4c3d71b had dropped the fix job's single concurrency group from alert triage's inherited rules. Codex caught it, and it is back, so overlapping triage runs cannot open two drafts for one fingerprint. The earlier Codex fixes and the simplify pass are otherwise unchanged. Scenario 4, the baseline section and the fix-job verdicts are byte-identical to main. `validate.sh --all` exits 0 and CI validate passes. |
|
@codex review Review current commit |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2ef9a192f7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The description now names alert triage, with a should-trigger case and a dedicated eval 4. An Alert is wrong draft carries its fingerprint, mutes repeat drafts, and falls back to a replayed known-bad sample when the alert has never caught a real failure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QS4mU5DxP81inR3AhxCcKy
|
@codex review Review current commit |
|
Codex Review: Didn't find any major issues. Keep them coming! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QS4mU5DxP81inR3AhxCcKy
|
@codex review Review current commit |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4c3d71b6ac
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QS4mU5DxP81inR3AhxCcKy
|
@codex review Review current commit |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 863d4d27e6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| ## Alert triage | ||
|
|
||
| A job where an agent reads a fired monitoring alert, with no person watching, and decides what it is before anything is fixed. An alert is not yet a failure, so triage has its own verdicts, parsed from a fixed first line like the fix job's; anything else counts as a real failure, so a bad parse can never silence an alert. The job runs under the fix job's other rules (read-only split from write, isolated settings, untrusted evidence, caps and dedupe in a single concurrency group), and every issue or draft it opens carries the alert's fingerprint, so an open or recently closed one mutes the next. |
There was a problem hiding this comment.
When this is implemented with GitHub Actions and three different alerts arrive while one triage run is active, putting every run in one concurrency group silently cancels the first pending run when the next one arrives, so an alert can receive no triage at all. GitHub documents that the default group retains only one pending run and replaces it with newer runs (GitHub Actions concurrency); require a non-dropping queue such as queue: max, or use fingerprint-scoped groups plus an atomic check-and-create barrier, rather than prescribing a single group without queuing.
Useful? React with 👍 / 👎.
1 participant
Summary
Task 685.
codebase-architecturenow says every incident or repeated mistake ends in a deterministic check. It also tells an unattended job how to triage an alert. Source: the 2026-10-08 call with Ben Friebe (R6, P6, R2, M-a, P9, M3, M4; plan ticket S40).references/enforcement-ladder.mdreferences/agent-runtime.md: a new Alert triage section after "Unattended fix jobs", with three verdicts.SKILL.md: line 118 names incidents and repeated mistakes. The description, the reference table row and Harden step 6 name alert triage.evals/evals.json: eval 3's first assertion now holds for a code fix and names the alert-definition exception. New eval 4 and a should-trigger case cover alert triage on its own.Decisions for the reviewer
erroroverwarn.evaluation-scenarios.mdis unchanged. None of its scenarios asserts a claim this PR touches. Alert triage gets its own eval inevals.jsoninstead.Stages: feature; skipped codebase-architecture (no module boundary), product-design/ui-* (no UI), blast-radius (about 40 lines of Markdown), swarm (one fresh-context reviewer instead).
Unchanged on purpose: "Never hand-roll a baseline", eval scenario 4, and the whole "Unattended fix jobs" section, including the verdicts at
agent-runtime.md:24. Not added, per the plan: a baseline or ratchet file, a debt-shrinks script, or a 400-line PR budget.Verification
skills/agent-skills-creator/scripts/validate.sh skills/codebase-architecture:0 FAIL, 31 PASS, 2 SKIP, exit 0.skills/agent-skills-creator/scripts/validate.sh --all: 0 FAIL on every skill, exit 0.python3 -m unittest discover -s maintenance/tests: OK.npx -y taste-lint@0.4.0 lint --dry-run --since origin/HEAD --profile instructions: PASS, no act-band findings.origin/main:origin/main.agent-runtime.mdchanges only line 3 plus an appended section../mac-mini/setup.sh skillsis re-run (#705).Open point: eval 3's prompt is about error issues, not alerts, yet its first assertion names the alert exception because the brief asks for that. Eval 4 now tests alert triage directly.
Learned: the riskiest part of an alert-definition exception is that "fix the threshold" quietly becomes "turn the alert off". The exception only stays narrow if the draft proves the alert would still have fired on the last real failure.
🤖 Generated with Claude Code
https://claude.ai/code/session_01QS4mU5DxP81inR3AhxCcKy