mutmut
mutmut copied to clipboard
boxed
Removed assumptions when mutant is killed
Commit 1ade5e5986ca69956d9cd5fecb3e71b264c73ed3 added the assumptions that if a mutant was killed regardless of the test file changing it'd still be marked as killed.
Even though it might offer a performance boost, I don't think it's really worth it because mutmut becomes blind to any kind of refactor, either in the source code or the test code.
Caught me offguard and I think it might happen to other developers too.
I don't agree with this change. It's not just a small performance boost, it's the difference between mutmut being something you can practically use, and not. If all kills are invalidated for each change to the test suite then you can't incrementally work towards your goal.
The reason for this PR was that a regression was introduced upon a refactor and mutmut didn't catch it, but the CI was able to do so.
I agree that a single change shouldn't invalidate the whole test suite, it'd be too much time consuming. Do you think it'd be worth to change the hash to work on a test file basis instead of suite?
The problem with that is that it's a big redesign of mutmut :)
Mutmut doesn't know anything about what tests cover what code. That's partly by design as it makes mutmut usable for many different test systems. It's also partly because of limitations in the tooling for figuring this out, and limitations in test selection.
I don't know if I've fully understood the underlying problem. Currently a change to the tests would trigger a rerun of the baseline tests:
https://github.com/boxed/mutmut/blob/5c31ba8e3eb5f1b5035c556a51a4c3327d4562da/mutmut/main.py#L459-L464
What was your situation? It sounds to me like the following since you say the CI was able to catch the regression:
- you started your mutation testing at a time where all tests were passing
- you stopped you mutation test run
- you modified the production code, but left the tests as they were
- you then continued your mutation test session, and mutmut did not notice? Instead, it probably marked every mutant as killed, as the tests were failing all the time?
In that case it might be a compromise to add an option to rerun the baseline tests first every time you resume your mutation testing session.
