Tests without mutations fail (possibly related to Java 9 Modules)

[turing85]

Reproducer:

• Checkout https://github.com/turing85/advent-of-code-2022/tree/pitest-bug:

  git clone [email protected]:turing85/advent-of-code-2022.git
  cd advent-of-code-2022
  git switch pitest-bug
  

• Build and run mutation tests on module day-21:

  ./mvnw \
    --projects day-21 \
    --define targetTests='de.turing85.advent.of.code.tttt.day.**'  \
    --activate-profiles disable-dependency-checker \
    clean \
    package \
    org.pitest:pitest-maven:mutationCoverage
  

• Observe that the unit tests pass
• Observe that all tests defined in MonkeyTreeTest and MonkeysParserTest fail without mutations.

Noteworthy:

• On all other modules, mutation testing works just fine.

• The project uses almost no external dependencies (maven plugins, lombok, junit5, google truth and pitest, including the junit5 plugin for pitest)

• The tests are all stateless, so no order problem

• The only difference between the failing module and all other modules is that the failing modules takes advantage of Java modules by

  • using interface MonkeyParser through the module-info.java,
  • implementing said interface with two classes, exposing the implementations through the module-info.java, and
  • loading the implementations of said interface through the ServiceLoader in MonkeysParser

  All tests in the aforementioned classes use the code related to the ServiceLoader. I suspect that the failing tests are related to this.

Logs of stdout and sterr are attached, run was done with --define verbose='true'.

[hcoles]

Yes, the issue is the java 9 module system. Before 9, service loader looked for a properties file to identify implementations, after 9 it also looks at module-info.

Pitest doesn't currently support module path. At some point it will have to, but a significant proportion of its user base is still stuck on java 8 and maintaining backwards compatibilty would be complicated if the build relied on java 9 features.

As a workaround you could add the legacy property file entries and then I believe the tests should run green.

[turing85]

Yes, the issue is the java 9 module system. Before 9, service loader looked for a properties file to identify implementations, after 9 it also looks at module-info.

Pitest doesn't currently support module path. At some point it will have to, but a significant proportion of its user base is still stuck on java 8 and maintaining backwards compatibilty would be complicated if the build relied on java 9 features.

Hm, I see. do we have this limitation documented somewhere?

As a workaround you could add the legacy property file entries and then I believe the tests should run green.

~~What exactly do you mean by this? Could you link to an example or a documentation page?~~ Found it (docs.oracle.com). And yes, you're right; this fixes the issue.

[verhasi]

I just ran into this issue also. I have checked the documentation/FAQ especially the "My tests normally run green but PIT says the suite isn’t green" section. Finally, I have figured it out and applied the same workaround as proposed here. Until you fix it, I suppose the FAQ could be extended with a short description.