Skip to content
Merged
5 changes: 5 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,11 @@ jobs:
# Validate skill/agent frontmatter and that every internal craft:<slug> reference resolves.
- name: Check skills & agents
run: node lib/check-skills.mjs
# Compares the SET OF OUTCOMES each review agent's two deliveries (agents/ vs opencode/agents/)
# can report, mapped from the rubric's words — not the two bodies in full, which can still
# diverge in guidance and detail with this gate green.
- name: Check delivery parity
run: node lib/check-delivery-parity.mjs
# The opencode delivery's agents/commands answer to a different frontmatter contract than the
# Claude Code ones above (no `name`, no pinned `model`, `mode: subagent`, `hidden: true`,
# `tools` a map with write/edit false), so it gets its own checker rather than a widened one.
Expand Down
4 changes: 3 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,7 @@ Every carrier below was observed by running it on 2026-08-31; none is inferred.
| "the manifests are valid" | the manifests as the official validator reads them | `npx --yes @anthropic-ai/claude-code plugin validate . --strict` | agent |
| "this skill/agent is well-formed and its `craft:` refs resolve" | the checker's verdict over `skills/`, `agents/`, `workflows/` | `node lib/check-skills.mjs` | agent |
| "this workflow script still parses in the sandbox" | the script compiled inside the sandbox wrapper | `node lib/check-workflows.mjs` | agent |
| "the two deliveries of a review agent can name the same set of outcomes in their bodies" | the set of outcomes each side's body can reach, per the checker's `OUTCOMES` table | `node lib/check-delivery-parity.mjs` | agent |
| "the helper logic is correct" | the test run | `node --test 'lib/**/*.test.mjs' 'opencode/**/*.test.mjs'` | agent |
| "the code is lint-clean" | ESLint over the linted scope | `npm run lint` — and read the raw exit code, not a wrapper's summary | agent |
| "the eval corpus is well-formed" | the checker's verdict | `node lib/check-evals.mjs` | agent |
Expand Down Expand Up @@ -154,6 +155,7 @@ Node.js 22 (CI pins `node-version: '22'`), plain ESM JavaScript. **No runtime de
| Syntax-check workflow scripts | `node lib/check-workflows.mjs` |
| Check skills and agents (frontmatter + `craft:<slug>` references) | `node lib/check-skills.mjs` |
| Check the evals corpus | `node lib/check-evals.mjs` |
| Check the two deliveries of each review agent report the same outcomes | `node lib/check-delivery-parity.mjs` |
| Validate plugin manifests | `claude plugin validate . --strict` |
| Lint | `npm run lint` (`eslint lib opencode/plugin --max-warnings 0`, identical to CI) |
| Typecheck the OpenCode plugin's TypeScript | `npm run check:types` (needs `npm ci --prefix opencode/plugin` once) |
Expand Down Expand Up @@ -184,7 +186,7 @@ No formatter. There **is** a typechecker now: `tsc --noEmit --strict` over `open
- **`lib/run-record.mjs` and `opencode/plugin/run-record.mjs` are NOT copies** — they are different modules sharing a filename (`lib` carries `countBySeverity`, `reviewVerdict`, the triage and fingerprint helpers; `opencode` carries `parseVerdict`, `buildAuditRecord`, `buildTriageRecord` and writes files directly). So editing "the other copy" edits the wrong file. **But they share exactly one export, `indexProjection`, and its two bodies already disagree** — verified 2026-09-03: the `lib` one emits `craftVersion`, `craftCommit`, `branch`, `head`, `round` and `outputTokens` and defaults `findingsTotal` to `0`; the OpenCode one emits none of the six and defaults to `null`. That is the trap in both directions: a column added to the index in `lib` is not added by the OpenCode writer, and lines from the two deliveries silently carry different shapes into one `index.jsonl`. No gate compares them.
- **The genuinely duplicated surfaces are `lib/run-logging.mjs`, `lib/run-record.mjs` and `lib/review-coverage.mjs`**, each inlined verbatim into the workflow engines inside `// >>> craft-inline` fences (`run-logging` and `run-record` into all four). That one IS gated: `node lib/check-workflows.mjs` byte-compares every region against the source and `--fix` regenerates them, so change the source and regenerate — never hand-edit a region. One exception, and it is deliberate: `review.js`'s `shq` sits **outside** the fence (excluded to avoid a duplicate declaration), so the checker is blind to it; a tripwire in `lib/review-coverage.test.mjs` compares it instead.
- **The OpenCode delivery has exactly one gate per surface, and no more.** `index.ts`, `orchestrator.ts`, `rust-audit.ts` and `triage-findings.ts` are compiled in CI by `tsc --noEmit --strict` (step "Typecheck opencode plugin") — but they remain outside ESLint (it globs `.js`/`.mjs`/`.cjs`, no TS parser configured) and outside `node --test`, so a type-clean file can still be wrong. `opencode/agents/*.md` and `opencode/commands/*.md` are shape-checked by `opencode/scripts/check-frontmatter.py` (step "Check opencode agent/command frontmatter"), which validates frontmatter shape only — it says nothing about whether a description triggers, and it never reads the body. Those two directories are also **not** in `lib/check-skills.mjs`'s foreign-plugin scan, which covers exactly `README.md`, `MAP.md`, `CLAUDE.md`, `opencode/README.md` and `opencode/install.sh` (plus `skills/`, `agents/`, `workflows/`): a `superpowers:` reference inside an OpenCode agent or command body is read by no checker at all. And `opencode/README.md` and `install.sh` are scanned for that and nothing else.
- **Nothing checks that the two deliveries agree — and that is the hole the gates do not close.** Every agent ships twice, once under `agents/` for Claude Code and once under `opencode/agents/`, and the two bodies are maintained by hand. No gate compares them, so they can diverge in *content* with CI fully green: the checkers above judge each file's shape in isolation, and shape is exactly what stays valid while meaning drifts apart. This is not hypothetical — as of 2026-09-02 on this branch, `agents/rust-reviewer.md` mentions `INCOMPLETE` 7 times and `opencode/agents/rust-reviewer.md` **0** times (it has no INCOMPLETE verdict path at all), the other three counterparts carry 2-3 against 6-7, and `agents/nix-reviewer.md` has no `opencode/agents/` counterpart whatsoever. Counted with `awk '{n+=gsub(/INCOMPLETE/,"")} END{print n+0}'` over each file. So: after editing an agent on either side, check the other side by hand — a green CI is not evidence that you did.
- **`node lib/check-delivery-parity.mjs` compares the SET OF OUTCOMES each agent's two deliveries (`agents/` and `opencode/agents/`) can report** — `ok`/`concern`/`bad`/`incomplete`, matched against the rubric's own words with the leading YAML frontmatter stripped so it reads the BODY, not the file (the checker's header comment carries the full rationale, including why frontmatter alone would mask a lost outcome). It does **not** compare guidance, rubric detail, or wording — those can diverge with this gate green. `agents/nix-reviewer.md` has no `opencode/agents/` counterpart by design (`UNPAIRED_BY_DESIGN` in the checker: the OpenCode delivery ships no Nix profile). Re-measured 2026-09-05 with `awk '{n+=gsub(/INCOMPLETE/,"")} END{print n+0}'` over each file: `rust-reviewer` 7 (root) vs 8 (opencode), `rust-architecture-reviewer` 7 vs 5, `rust-security-scanner` 6 vs 4, `rust-miri` 6 vs 6 — as of that measurement, the token appears on both sides of all four pairs, but a raw count is not what the gate checks (see the checker's comment for why). After editing an agent on either side, still check the other side by hand for meaning — the gate proves the outcome set matches, not that the two bodies say the same thing.
- **The two agent frontmatter contracts are genuinely different, hence two checkers.** Root `agents/*.md` (Claude Code) require `name` matching the filename, a pinned `model`, and `tools` as a list. `opencode/agents/*.md` require the opposite on two of those: no `name` (identity comes from the filename) and **no** `model` (an OpenCode sub-agent must inherit the session model), plus `mode: subagent`, `hidden: true`, and `tools` as a *map* with `write`/`edit` false. Widening `lib/check-skills.mjs` to cover both would mean two rule sets behind one entry point; the existing Python script already encodes the OpenCode one.
- **`opencode/plugin/` pins its runtime dependencies exactly and commits `package-lock.json`.** `@opencode-ai/plugin` and `@opencode-ai/sdk` were `"*"` — any upstream release could break CI at an arbitrary moment. CI installs them with `npm ci --prefix opencode/plugin`. This closure is deliberately separate from the root `package.json`, which stays devDependencies-only so the shipped plugin installs with nothing to fetch. The toolchain is pinned exactly too (`typescript` `5.9.3`, `@types/node` `25.9.5`, not ranges): `npm ci` would honour the lockfile either way, but under a range an `npm install` here can pull a newer `tsc` whose new checks fail CI for reasons unrelated to the diff. Bumping either is then a visible, deliberate commit.
- **Workflow scripts cannot be checked with `node --check`** — they have top-level `export` + `await` + `return`. `lib/check-workflows.mjs` reproduces the sandbox wrapper and compiles each one; when you change the shape of a workflow script, verify the wrapper still accepts it.
Expand Down
Loading
Loading