hcoles
Inconsistent coverage when running JUnit tests in parallel
After a single mvn clean install invocation, I get inconsistent coverage with subsequence calls to mvn org.pitest:pitest-maven:mutationCoverage
JDK target: 8
$ for x in (seq 1 10); mvn org.pitest:pitest-maven:mutationCoverage 2>/dev/null | grep 'Generated 7 mutations Killed'; sleep 2; end
>> Generated 7 mutations Killed 6 (86%)
>> Generated 7 mutations Killed 3 (43%)
>> Generated 7 mutations Killed 6 (86%)
>> Generated 7 mutations Killed 6 (86%)
>> Generated 7 mutations Killed 6 (86%)
>> Generated 7 mutations Killed 6 (86%)
>> Generated 7 mutations Killed 6 (86%)
>> Generated 7 mutations Killed 6 (86%)
>> Generated 7 mutations Killed 6 (86%)
>> Generated 7 mutations Killed 6 (86%)
JDK target: 14
$ for x in (seq 1 10); mvn org.pitest:pitest-maven:mutationCoverage 2>/dev/null | grep 'Generated 7 mutations Killed'; sleep 2; end
>> Generated 7 mutations Killed 6 (86%)
>> Generated 7 mutations Killed 6 (86%)
>> Generated 7 mutations Killed 6 (86%)
>> Generated 7 mutations Killed 6 (86%)
>> Generated 7 mutations Killed 6 (86%)
>> Generated 7 mutations Killed 6 (86%)
>> Generated 7 mutations Killed 7 (100%)
>> Generated 7 mutations Killed 6 (86%)
>> Generated 7 mutations Killed 3 (43%)
>> Generated 7 mutations Killed 3 (43%)
Something to be noticed
It happens less setting up the target to JDK 8, it happens for JDK 11 as frequently as for JDK 14.
Work around (which is the default for JUnit5):
junit.jupiter.execution.parallel.enabled=false
The source code: src.zip
I have observed the same, and indeed it seems that PIT is not compatible with junit.jupiter.execution.parallel.enabled = true Eventually it would be nice that it turn off junit.jupiter.execution.parallel.enabled automatically if detected when using pitest-junit5-plugin.
It does so since version 1.1.1, but it would be nicer if parallel execution would be supported properly. Assuming the same problems will hit with Spock in parallel mode.
While pitest/pitest-junit5-plugin#74 already worked-around this specifically for Jupiter engine, pitest/pitest-junit5-plugin#91 provides a generic fix that should work with all test engines and was also tested with Spock. So latest when pitest/pitest-junit5-plugin#91 gets merged, this issue can be closed.