fix: keep the mutation timeout window independent of sharding - #41
Open
maks-oleksyuk wants to merge 3 commits into
Open
fix: keep the mutation timeout window independent of sharding#41maks-oleksyuk wants to merge 3 commits into
maks-oleksyuk wants to merge 3 commits into
Conversation
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Aug 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The timeout given to each mutation is derived from the duration of the initial test run. A shard runs only part of the suite, so its initial run is shorter and the window shrinks, while the work a single mutation does is unchanged: the subprocess still pays the framework bootstrap plus the tests covering that mutation. Mutations killed on a full run get cut off as timeouts instead, and since a timeout counts as a killed mutation the score stays high while the run gets slower and the result less honest.
What changed
--update-shardscannot be combined with--shard, so the run that writes the timings file is always unsharded and can serve as the reference. It records the duration of its initial test run as metadata next to the units, and a sharded run takes whichever is longer, its own measurement or that reference. An unsharded run behaves exactly as before.The window is deliberately still derived from the suite rather than from the tests covering each individual mutation. The narrower model is more accurate, but it would shorten the window wherever the covering tests are fast, which risks introducing timeouts in suites that pass today.
Result
Measured on 4 shards: 8 timeouts instead of 18, critical path 105s instead of 135s, total work across the shards 364s instead of 483s.
Adding up the shards gives
tested=661 untested=2 timeout=8 total=671 score=99.70%againsttested=664 untested=2 timeout=5 total=671 score=99.70%for an unsharded run of the same code. Total, escaped and score match exactly; the remaining difference between 8 and 5 timeouts sits inside the run-to-run variance of the machine.Depends on
Requires
Shard::metadata()added in pestphp/pest#1829.This branch also carries the commits from #39 and #40, which should be merged first; once they are, this diff reduces to the timeout change alone.