Skip to content

Commit 8b92814

Browse files
donislawdevclaude
donislawdev
and
claude
authored
ci: mutation testing in the repository, weekly and on pull requests (#193)
* ci: mutation testing in the repository, weekly and on pull requests The mutation runner and its 1786 entries move from a local directory into internal/guard/mutation, and a new workflow runs them: every entry each Sunday, cut into sixteen parts that run side by side, and on a pull request only the entries the change reaches - the file an entry mutates, the file its guard lives in, or the entry itself. Pull requests get warnings for now, the weekly run goes red. What changed in the runner to make that possible: - the root and the go command are found rather than written down as paths on one Windows machine - worker trees keep one path between runs and are built before anything is timed. A tree under a fresh name built the guard binary in 409.7 s against 4.8 s warm, and the first entry of every worker paid that inside its own time ceiling - an entry on a file this platform does not build, a file the checkout does not have, or a guard that does not exist here is UNJUDGED with the reason, rather than a hole in a healthy guard - --changed, --shard, --plan and --json, a canary that checks the selection before every plan, and report.py, which treats a part that never reported as a failure rather than as a smaller run The coverage guard reads the tracked list and fails when it is missing. It used to return early on every CI run, and eight guards gathered without an entry in that time. Seven new entries and one re-pointed close them, proven 9 of 9 locally. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * mutation: a failed worker, a leftover tree directory and the disk check, from the review of #193 - A worker whose entry raised outside the verdicts judge knows - a rebuild that could not delete or copy, a process that did not start - died with the exception and left the entry without a verdict. The entry is now DAMAGED with the error, that worker stops, the others go on, and dispatch says how many entries were never judged when every worker has stopped. - A tree directory the sweep could not remove is refused in words naming what is left, rather than with the traceback of a mkdir. A failure to make the trees is written to the JSON report as a stop. - The disk check asks about the volumes a run writes to - the temporary directory for the worker trees and GOCACHE for the build cache - rather than the volume of the checkout, and names the volume it refuses on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent ae19084 commit 8b92814

12 files changed

Lines changed: 17150 additions & 27 deletions

File tree

‎.github/requirements-mutation.in‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
# What the mutation runner needs beyond Python itself, pinned, in a file a bot
2+
# can edit.
3+
#
4+
# The runner is internal/guard/mutation/mutate.py, and CI runs it from
5+
# .github/workflows/mutation.yml. It uses psutil for one thing: watching the
6+
# memory of the whole process tree an entry starts, so that a mutation which
7+
# removes a limit cannot take the machine down with it. It is never linked into
8+
# either binary and go.mod never sees it.
9+
#
10+
# Licence read from the pinned version rather than recalled, on 2026-10-15:
11+
# BSD-3-Clause, stated by the wheel metadata and carried in full in the LICENSE
12+
# file beside it.
13+
#
14+
# requirements-mutation.txt next to this file is compiled from it by
15+
# pip-compile, the same way requirements-semgrep.txt is, so Dependabot proposes
16+
# a set pip can install. To compile it by hand, from the root of the
17+
# repository:
18+
#
19+
# docker run --rm -v "$PWD/.github:/w" -w /w python:3.14-slim sh -c "pip install -q pip==26.1.1 pip-tools==7.5.3 && pip-compile --strip-extras --no-emit-index-url --output-file=requirements-mutation.txt requirements-mutation.in"
20+
psutil==7.2.2

‎.github/requirements-mutation.txt‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
#
2+
# This file is autogenerated by pip-compile with Python 3.14
3+
# by the following command:
4+
#
5+
# pip-compile --no-emit-index-url --no-index --output-file=requirements-mutation.txt --strip-extras requirements-mutation.in
6+
#
7+
psutil==7.2.2
8+
# via -r requirements-mutation.in

‎.github/workflows/mutation.yml‎

Lines changed: 171 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,171 @@
1+
# Mutation testing: every guard broken on purpose, to see that it notices.
2+
#
3+
# A green suite says the code does what the guards check. It does not say the
4+
# guards check anything - that sentence needs each one broken on purpose, and
5+
# internal/guard/mutation/entries.py holds one entry per break. Until
6+
# 2026-10-15 the entries lived outside the repository and ran on one machine,
7+
# where a full pass took over an hour and was rarely run. Of 1779 entries that
8+
# day, 236 had been proven against the code as it stood.
9+
#
10+
# Two kinds of run, one path through the same jobs:
11+
#
12+
# - weekly, and by hand from the Actions tab: every entry, cut into sixteen
13+
# parts that run side by side. Red when a mutation is not caught. This is
14+
# what catches a guard that stopped reaching its code through a change in
15+
# a third file, which no selection by file can see.
16+
# - on a pull request: only the entries the change reaches - the file an
17+
# entry mutates, the file its guard is in, or the entry itself. Reported
18+
# and never red for now, while the runner earns trust on these machines.
19+
#
20+
# The plan job decides how many entries and parts there are, each part judges
21+
# its own share in disposable copies of the tree, and the report job puts the
22+
# parts together. A part that never reports is a failure there rather than a
23+
# smaller run.
24+
name: mutation
25+
26+
on:
27+
pull_request:
28+
schedule:
29+
# Sunday night. Monday morning already holds the fuzzing job of ci.yml at
30+
# 04:17, the OSV scan at 05:17 and the tool versions at 06:00, and this
31+
# run takes sixteen of the twenty jobs a free plan runs at once.
32+
- cron: "37 1 * * 0"
33+
workflow_dispatch:
34+
35+
permissions:
36+
contents: read
37+
38+
# A newer push to the same pull request makes the older run worthless, so it
39+
# is cancelled rather than left to hold sixteen machines.
40+
concurrency:
41+
group: mutation-${{ github.event.pull_request.number || github.ref }}
42+
cancel-in-progress: true
43+
44+
env:
45+
GO_VERSION: "1.27.2"
46+
# Built as the test job builds on Linux, so a verdict here is about the
47+
# guards CI actually runs. Entries on files only a cgo build or only
48+
# Windows compiles come back UNJUDGED, said out loud in the report.
49+
CGO_ENABLED: "0"
50+
51+
jobs:
52+
plan:
53+
name: which entries
54+
runs-on: ubuntu-latest
55+
timeout-minutes: 10
56+
outputs:
57+
entries: ${{ steps.plan.outputs.entries }}
58+
parts: ${{ steps.plan.outputs.parts }}
59+
shards: ${{ steps.plan.outputs.shards }}
60+
steps:
61+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
62+
with:
63+
persist-credentials: false
64+
# The whole history, because a pull request is compared with its
65+
# base and the base has to be here to be compared with.
66+
fetch-depth: 0
67+
68+
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
69+
with:
70+
python-version: "3.14"
71+
72+
- name: what the runner needs
73+
run: pip install -r .github/requirements-mutation.txt
74+
75+
- name: the plan
76+
id: plan
77+
env:
78+
BASE: ${{ github.event_name == 'pull_request' && github.event.pull_request.base.sha || '' }}
79+
run: |
80+
python internal/guard/mutation/mutate.py --plan ${BASE:+--changed "$BASE"} >> "$GITHUB_OUTPUT"
81+
if grep -qx 'entries=0' "$GITHUB_OUTPUT"; then
82+
echo "No mutation entry is reached by this change, so none was run." >> "$GITHUB_STEP_SUMMARY"
83+
fi
84+
85+
run:
86+
name: part ${{ matrix.shard }}
87+
needs: plan
88+
if: needs.plan.outputs.entries != '0'
89+
runs-on: ubuntu-latest
90+
# A part of the weekly run is about 110 entries at three at a time. The
91+
# first run measures what that costs here, and this ceiling is far above
92+
# it so that a hang ends as a part that did not report.
93+
timeout-minutes: 120
94+
strategy:
95+
fail-fast: false
96+
matrix:
97+
shard: ${{ fromJSON(needs.plan.outputs.shards) }}
98+
steps:
99+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
100+
with:
101+
persist-credentials: false
102+
fetch-depth: 0
103+
104+
- uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
105+
with:
106+
go-version: ${{ env.GO_VERSION }}
107+
check-latest: false
108+
cache: true
109+
110+
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
111+
with:
112+
python-version: "3.14"
113+
114+
- name: what the runner needs
115+
run: pip install -r .github/requirements-mutation.txt
116+
117+
- name: the entries of this part
118+
# Never red on its own. The verdicts go to the JSON file and the
119+
# report job decides - and a step that crashed before writing it is
120+
# found there as a part that did not report.
121+
continue-on-error: true
122+
env:
123+
BASE: ${{ github.event_name == 'pull_request' && github.event.pull_request.base.sha || '' }}
124+
SHARD: ${{ matrix.shard }}
125+
PARTS: ${{ needs.plan.outputs.parts }}
126+
run: >
127+
python internal/guard/mutation/mutate.py ${BASE:+--changed "$BASE"}
128+
--shard "$SHARD/$PARTS" --workers 3 --json "result-$SHARD.json"
129+
130+
- uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
131+
if: always()
132+
with:
133+
name: mutation-result-${{ matrix.shard }}
134+
path: result-*.json
135+
if-no-files-found: ignore
136+
retention-days: 14
137+
138+
report:
139+
name: what the run found
140+
needs: [plan, run]
141+
if: always() && needs.plan.result == 'success' && needs.plan.outputs.entries != '0'
142+
runs-on: ubuntu-latest
143+
timeout-minutes: 10
144+
steps:
145+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
146+
with:
147+
persist-credentials: false
148+
149+
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
150+
with:
151+
python-version: "3.14"
152+
153+
- uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
154+
# A run in which no part reported has nothing to download, and that
155+
# is for the report to say rather than for this step to fail on.
156+
continue-on-error: true
157+
with:
158+
pattern: mutation-result-*
159+
path: results
160+
merge-multiple: true
161+
162+
- name: the verdict
163+
env:
164+
ENTRIES: ${{ needs.plan.outputs.entries }}
165+
PARTS: ${{ needs.plan.outputs.parts }}
166+
# Red on the weekly run and on a run by hand. On a pull request the
167+
# same findings are warnings until the runner has earned trust here.
168+
GATE: ${{ github.event_name != 'pull_request' && '--gate' || '' }}
169+
run: >
170+
python internal/guard/mutation/report.py --dir results
171+
--entries "$ENTRIES" --shards "$PARTS" $GATE

‎CONTRIBUTING.md‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -109,7 +109,11 @@ whose test suites depend on it.
109109
library will not do.
110110
- **Every change in behaviour comes with a test that fails without it.** A green
111111
suite is not evidence on its own. The question to answer in the pull request
112-
is "which test would go red if this change were undone".
112+
is "which test would go red if this change were undone". Each test in
113+
`internal/guard` is also broken on purpose by an entry in
114+
`internal/guard/mutation/entries.py`, and the `mutation` check on a pull
115+
request runs the entries the change reaches. A new test needs an entry there,
116+
or a line in `internal/guard/mutationcoverage_test.go` saying why it has none.
113117
- **Words a user reads are English, with a flat hyphen and no semicolons.** That
114118
covers the README, the changelog, `--help`, every message, and every comment
115119
in the code. A test enforces the punctuation.
@@ -123,7 +127,9 @@ whose test suites depend on it.
123127
- Keep it to one subject. A pull request that fixes a bug and tidies three
124128
unrelated files is one that cannot be reviewed or reverted cleanly.
125129
- CI has to be green. It builds and tests on Windows, Linux and macOS, runs the
126-
linters, and checks the dependency and supply chain gates.
130+
linters, and checks the dependency and supply chain gates. The `mutation`
131+
check reports a break no test noticed as a warning and does not block the
132+
merge for now.
127133
- The commit message and the pull request are public and permanent. Do not put
128134
anything in them you would not show a stranger.
129135
- Pull requests are merged by the maintainer, as a squash.

0 commit comments

Comments
 (0)