hcoles
Should say in documentation that Pitest is deterministic
First of all, the idea of Pitest is from a genius! Without any additional code, you increase the quality of the tests and point new cases for tests.
It's very important to known that Pitest is deterministic or not, since it uses the concept of mutation. The full meaning of mutation is directly related to randomness.
I found a comment from the author saying that "Pitest is fully deterministic": https://github.com/hcoles/pitest/issues/384
We started to use it in our CI and I hope that it's deterministic, that each CI work tests the same thing.
Best regards.
Is it random?
No.
Given the same input pitest will always generate the same mutants, and (with a couple of caveats) will always produce the same results.
Pitest works hard to be fully deterministic, but two factors might cause the results to differ slightly between two runs with the same input.
Timeouts
Mutants causing infinite loops are detected by comparing the time taken to run a test without the mutant to the time taken when the mutant is present. Both these measurements can be affected by external factors (other processes on the machine etc etc), so a mutant may be detected as timed out on one run, but killed or surviving on another.
Static initializers
As discussed above static initalization code causes some problems for mutation testing, in certain circumstances it can also esult in small differences between runs, especially if timeouts occurs as these require starting a new jvm.
https://pitest.org/faq/
You should be fine, as long as you don't change PIT's configs (timeouts and mutators)
