Skip to content

fix: keep the mutation timeout window independent of sharding - #41

Open
maks-oleksyuk wants to merge 3 commits into
pestphp:5.xfrom
maks-oleksyuk:pr/mutation-shard-timeout
Open

fix: keep the mutation timeout window independent of sharding#41
maks-oleksyuk wants to merge 3 commits into
pestphp:5.xfrom
maks-oleksyuk:pr/mutation-shard-timeout

Conversation

@maks-oleksyuk

Copy link
Copy Markdown

The timeout given to each mutation is derived from the duration of the initial test run. A shard runs only part of the suite, so its initial run is shorter and the window shrinks, while the work a single mutation does is unchanged: the subprocess still pays the framework bootstrap plus the tests covering that mutation. Mutations killed on a full run get cut off as timeouts instead, and since a timeout counts as a killed mutation the score stays high while the run gets slower and the result less honest.

What changed

--update-shards cannot be combined with --shard, so the run that writes the timings file is always unsharded and can serve as the reference. It records the duration of its initial test run as metadata next to the units, and a sharded run takes whichever is longer, its own measurement or that reference. An unsharded run behaves exactly as before.

The window is deliberately still derived from the suite rather than from the tests covering each individual mutation. The narrower model is more accurate, but it would shorten the window wherever the covering tests are fast, which risks introducing timeouts in suites that pass today.

Result

Measured on 4 shards: 8 timeouts instead of 18, critical path 105s instead of 135s, total work across the shards 364s instead of 483s.

Adding up the shards gives tested=661 untested=2 timeout=8 total=671 score=99.70% against tested=664 untested=2 timeout=5 total=671 score=99.70% for an unsharded run of the same code. Total, escaped and score match exactly; the remaining difference between 8 and 5 timeouts sits inside the run-to-run variance of the machine.

Depends on

Requires Shard::metadata() added in pestphp/pest#1829.

This branch also carries the commits from #39 and #40, which should be merged first; once they are, this diff reduces to the timeout change alone.

maks-oleksyuk and others added 3 commits August 7, 2026 14:59
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant