Detect nondeterministic tests before a mutation run

[adriendellagaspera]

cargo-mutants assumes the test suite is deterministic. In practice it's often not: unseeded proptest/quickcheck or rand::random(), tests binding a fixed port or using wall-clock time... With this, the same mutant can show MISSED on one run and CAUGHT on the next, and you only notice after a full run, which can take long.

Idea: these tests are always fixable upstream, and cargo-mutants could help catch them (cheaply). A preflight check (like cargo mutants doctor, or a flag) could scan for these patterns before running, and warn or refuse to run.
It could be static (grep/AST for proptest! without a seed, rand::random, fixed-port binds etc.), or a runtime check (run the baseline suite twice and diff the results), or both.

[sourcefrog]

I think you're basically talking about flaky tests: tests where the outcome isn't deterministic from the source tree.

You're quite right they'll cause problems for cargo-mutants, although of course they also cause problems for any other use of the test suite.

I think it's at least worth warning about them in the manual, or perhaps I already did.

I'm not sure about building flaky test detection into cargo-mutants:

• This seems only weakly related to cargo-mutants: people might want to find flakes in many other cases
• It seems complicated in general: looking for literal use of random numbers is fine, but there can be so many other causes including concurrency.

Perhaps there's value in a simple "retry failures n times" like in Bazel, but that would make things even slower, and presumably most tests are not flaky...

Worth thinking about anyhow.

[adriendellagaspera]

My point is not really flaky tests actually. proptest-style tests pass deterministically on correct code, but inputs change with the seed. Baseline never flakes, mutation coverage can: a mutant getting caught just means that run's random inputs happened to hit the mutated branch. Call that "mutation-flakiness".

Idea: on a MISSED, rerun once before reporting it. If the verdict flips (regardless of why, RNG seed or anything else timing-dependent), flag it as unstable instead of a flat MISSED. Only applies to MISSED, so cost stays marginal on a suite that's already well covered.

On top of that, pinning one seed per invocation (generated or given, exported as PROPTEST_RNG_SEED etc. to baseline and every mutant) makes a flagged case reproducible so you can dig into it. You'd have a debugging aid for proptest-specific cases.

Clearly, this mostly matters for projects that gate CI on zero surviving mutants and lean on property-based tests for coverage.

[sourcefrog]

I do somewhat talk about this in the manual:

cargo-mutants can only help if the test suite is hermetic: if the tests are flaky or non-deterministic, or depend on external state, it will draw the wrong conclusions about whether the tests caught a bug.

That's not to say either the manual or the behavior cannot be improved. In particular it might be good to suggest using a predictable RND seed, or perhaps we should do that by default.

However, if the mutant is missed, it's telling you "your test suite wouldn't catch this bug." If the testsuite is non-hermetic, then that statement could be weakened to "your test suite might not catch this bug," but it's still something that you probably want to look in to.

Clearly, this mostly matters for projects that gate CI on zero surviving mutants and lean on property-based tests for coverage.

If the proptest suite isn't enough to reliably catch some class of important bugs then, possibly, it's a sign that you need more deterministic tests and proptest should be only a second line of defense? Because, all these same bugs or regressions could at least in principle occur through organic changes to the program. Although I can also imagine how developers of some programs might heavily on proptest and think it would be uneconomic to write specific tests.

Idea: on a MISSED, rerun once before reporting it. If the verdict flips (regardless of why, RNG seed or anything else timing-dependent), flag it as unstable instead of a flat MISSED. Only applies to MISSED, so cost stays marginal on a suite that's already well covered.

Why is twice enough, though? Even if proptest catches the mutation 50% of the time, we're only at a 75% chance of correctly catching it on two attempts?

Perhaps some other advice we could give is:

• If you rely heavily on proptest and some mutants are missed, and you feel that proptest would have caught them given enough attempts, consider turning up PROPTEST_CASES. Consider doing this not only when running cargo-mutants but also when running tests altogether, because it's telling you that proptest is capable of catching interesting bugs, but the default number of cases is not enough to explore the input space.
• If proptest catches mutations only after many attempts, consider either committing the failing seed or writing a specific focused test case for the bug.
• Set PROPTEST_RNG_SEED to a single value per cargo-mutants run so that you can reproduce the tests.

Is this something you encountered using cargo-mutants on an existing tree? Is that tree available and/or can you tell me a bit more about what its tests are like and what mutants were missed?