- A vague requirement went undetected while a contradiction in the same workspace was caught — NON-CRITICAL
Uploaded workspace, edited to introduce defects. Two defects were added to the completed Astro Data Search design:
output.md — a "Recommended dataset" section telling the agent to name the single best choice, which contradicts scope.md and guardrails/no-best-dataset-metadata-proxy.md
scope.md — a Performance section reading "Searches should complete quickly so the researcher is not left waiting", with no threshold or number
Running /self-review all caught the "best dataset" contradiction — quoted both sides with line numbers and graded it [critical] — but did not flag the "complete quickly" statement at all, despite covering the whole workspace and quoting scope.md for other purposes.
Steps: append an unquantified requirement to scope.md, run /self-review all, check whether it appears under Unclear.
Also seen in the walkthrough interview workspace. A success criterion was given as "Turnaround should be fast enough that it doesn't hold up a demonstration". The automatic end-of-stage review did not flag it; the manual /self-review immediately afterwards did, as Unclear [nice-to-have].
- Review returns a permission error and nothing else — NON-CRITICAL
/self-review returns only Permission denied for read on 'guardrails/no-secrets-or-credentials.md': Protect sensitive files. No review is produced. Re-sending the command generates the review normally.
Steps: upload project artifact files (one with "secret" in the filename) → new chat → /self-review as the first message → error → re-send the command, review is generated.
Marked non-critical: reaching this requires uploading a prepared workspace, which is a developer path rather than something an end user would do.
Uploaded workspace, edited to introduce defects. Two defects were added to the completed Astro Data Search design:
output.md — a "Recommended dataset" section telling the agent to name the single best choice, which contradicts scope.md and guardrails/no-best-dataset-metadata-proxy.md
scope.md — a Performance section reading "Searches should complete quickly so the researcher is not left waiting", with no threshold or number
Running /self-review all caught the "best dataset" contradiction — quoted both sides with line numbers and graded it [critical] — but did not flag the "complete quickly" statement at all, despite covering the whole workspace and quoting scope.md for other purposes.
Steps: append an unquantified requirement to scope.md, run /self-review all, check whether it appears under Unclear.
Also seen in the walkthrough interview workspace. A success criterion was given as "Turnaround should be fast enough that it doesn't hold up a demonstration". The automatic end-of-stage review did not flag it; the manual /self-review immediately afterwards did, as Unclear [nice-to-have].
/self-review returns only Permission denied for read on 'guardrails/no-secrets-or-credentials.md': Protect sensitive files. No review is produced. Re-sending the command generates the review normally.
Steps: upload project artifact files (one with "secret" in the filename) → new chat → /self-review as the first message → error → re-send the command, review is generated.
Marked non-critical: reaching this requires uploading a prepared workspace, which is a developer path rather than something an end user would do.