Mutation score is changing and not stable enough

[vmassol]

Hi, I've updated pitest/descartes to use descartes 1.2, pitest 1.4.0 and pitest-junit5-plugin 0.5 and I've found today that plenty of mutation scores are changing (without any change to the sources). I don't know why but it's looking bad.

Somehow I have the feeling it could be related to the introduction of pitest-junit5-plugin in https://github.com/xwiki/xwiki-commons/commit/0a7d7eea2d4607d43a42f5133bc8291be6542104

Some examples:

• https://github.com/xwiki/xwiki-commons/commit/88f8b64ce5b5792c1b1994bdc3060345a02225fa
• https://github.com/xwiki/xwiki-commons/commit/85c9c98424552f58c13ff6be4d119c761d534c9f
• https://github.com/xwiki/xwiki-commons/commit/574c33311ddb20de76dbe9dc8c956e4a6bba6f9c
• https://github.com/xwiki/xwiki-commons/commit/f5394b424a0136854a771d6d5d8fee6fcb7f76d3
• https://github.com/xwiki/xwiki-commons/commit/2032022c6f8f705be1811f1014726f5f597f4abb

Any idea?

[vmassol]

One guess is that timeouts are in play here and when there's a timeout, it's counted as a bad score.

We could increase the timeout I guess but that'll also increase a lot the build time (we build all XWiki with descartes).

[oscarlvp]

When PIT performs the mutation analysis it multiplies by a factor the timeout considering the original execution time. By default this factor is 1.25. This timeout could be configured to be used only in the mutation analysis phase instead of changing the original timeout. You can check the timeoutFactor property. See here.

[surli]

Hi @oscarlvp

we recently configured some threshold to detect mutation score regression, but we got really stuck with those timeouts even when setting the timeoutFactor.

For example, our build spotted a mutation score of 80 against 88, that I was not able to reproduce on my own computer. When setting the timeoutFactor to 10 on an offline agent on the CI, I got successively the following mutation scores: 90, 88, 83, 85 and 88 again.

It's really hard to stabilize a threshold in this case. Any advice how we could improve that? Is there a limit value for the timeoutFactor? Should we use a timeoutConstant instead?

[vmassol]

FTR here's what we did with the timeoutFactor: https://github.com/xwiki/xwiki-commons/commit/10dd48f8ca7e7bf024216d9d876abb0e6fb71950

[vmassol]

Recent case of timeout: https://github.com/xwiki/xwiki-platform/pull/946/files

[oscarlvp]

Report for xwiki/xwiki-commons@88f8b64 included here Test are probably unstable when using the mutated values. xwiki/xwiki-commons@574c333 corresponds to the same issue.

[oscarlvp]

Report for xwiki/xwiki-commons@85c9c98 included here The mutation score was stable after 100 executions. No mutant is killed by a timeout. @vmassol @surli How was the first score determined for this module?

[surli]

I guess you meant https://github.com/xwiki/xwiki-commons/commit/85c9c98424552f58c13ff6be4d119c761d534c9f (I got a 404 with your link).

AFAIK we computed the score by running the following command line and going to the pitest report:

mvn clean install -Pquality -Dxwiki.pitest.skip=false -Djacoco.skip=true -Dxwiki.checkstyle.skip=true

[oscarlvp]

@surli This is the same way I'm computing the score to reproduce the issues. However, there is no change for this particular commit and module over 100 executions. So I'm wondering, if by any chance, there was a mistake with the first score. Is the issue still happening on your side?

[surli]

I don't know for this module maybe @vmassol has more information about that one. Now I know that for https://github.com/xwiki/xwiki-platform/pull/946/files even executing it dozens of time on the same machine I didn't get a change. It's when executing it on another machine with other spec (less memory / processor) that I spotted the difference.

[oscarlvp]

As for xwiki/xwiki-commons@2032022 the analysis produces a 0 mutation score as no mutant is reported as covered. There is a configuration problem. xwiki-commons is using the Junit 5 PIT plugin. For this particular module Junit 5 is not in the classpath, so the plugin fails to execute and therefore finds no test class. fixing the configuration should fix the issue. Detailed report here

[oscarlvp]

@vmassol @surli There is indeed a difference on how the initial scores were computed. PIT was configured in xwiki/xwiki-commons@cb0ff74 to use Junit 4. In xwiki/xwiki-commons@0a7d7ee the configuration switched to Junit 5. This was done on 22-05-2018. The same day were reported significant differences in the score, which are the ones listed in the initial description of this issue. Here is a summary of the issues and the outcome when trying to reproduce it.

Commit	Score change	Outcome
xwiki/xwiki-commons@88f8b64	from 68 to 40	Score unstable but with no significant change. Might be due to some tuned test cases.
xwiki/xwiki-commons@85c9c98	from 67 to 36	Score stable at 36% after 100 executions.
xwiki/xwiki-commons@574c333	from 40 to 39	Same as xwiki/xwiki-commons@88f8b64
xwiki/xwiki-commons@f5394b4	from 21 to 11	Score stable at 11% after 100 executions.
xwiki/xwiki-commons@2032022	from 20 to 0	Score stable at 0% (no coverage) after 30 executions.

As hinted by the configuration issue for xwiki/xwiki-commons@2032022 the drastic score change could be explained by the change of test plugin for PIT from Junit 4 to Junit 5. The Junit 5 plugin is not as stable as the other and may be missing some tests. xwiki/xwiki-commons@2032022 should be solved by adding Junit 5 to the module configuration. The actual unstable score from xwiki/xwiki-commons@88f8b64 and xwiki/xwiki-commons@574c333 should be solved by removing the test cases identified as problematic (entropy generation).

[surli]

Thanks for this report @oscarlvp ! To continue on this topic, we got this weekend a report from our pit-descartes CI job with this:

xwiki-platform-webjars-api: Mutation score of 22 is below threshold of 24

The job is now back to normal. What's interesting is that nothing has been committed this weekend on any of our repo on master, and we didn't change anything in our config. So it does prove that we still experiencing this timeout issue. In order to reproduce the state of the three repo used for building at that moment were the following: https://github.com/xwiki/xwiki-commons/commit/12c32031e82de966018b1bfe65bbca67ea0d3d18 https://github.com/xwiki/xwiki-rendering/commit/2a8a41b4533276091007923d4d338ac42eae45c8 and https://github.com/xwiki/xwiki-platform/commit/d2b8e20817236f2ca04de4b4c0246e765efd38cf

You can find the job report here on our CI

[oscarlvp]

You're welcome @surli. I haven't checked the xwiki-platform issues you have reported. I started with the issues from xwiki-commons. Given the small percentage change it is likely to be a timeout issue (in the same sense as xwiki/xwiki-commons@88f8b64). I will do the same experiment with the commit your are reporting.

[oscarlvp]

@surli I can't build xwiki-platforms on my side so I inspected the console output from Jenkins and there is indeed a variation. The score for xwiki-platform-webjars-api is 11/45 -> 24 % for all builds except for the following 4:

• 132 10/45 -> 22%
• 146 12/45 -> 27%
• 147 10/45 -> 22%
• 148 12/45 -> 27%

From the details in the console output it can be seen that the variation concerns two mutants: a true mutant, affecting one boolean method and a void mutant affecting of course a void method. No mutant is killed with a timeout, so the timeout factor will not solve this particular case. Is it possible to get the report files generated by PIT during those builds? That would shed a light on which methods are being affected by those mutants and the tests involved.

[surli]

Is it possible to get the report files generated by PIT during those builds?

I don't think it is we run it with a mvn clean goal and the workspaces are not archived. But the last report is available in the target directory.

[vmassol]

You can get the last build's workspace from https://ci.xwiki.org/job/xwiki-platform_pitest/ws/xwiki-platform-core/xwiki-platform-webjars/xwiki-platform-webjars-api/target/pit-reports/

[oscarlvp]

Thanks @vmassol The mutations.json file in the workspace is actually showing some TIMED_OUT mutants. One of them is void. Of the three true mutants one is marked as SURVIVED. Both these mutants are covered by the following test case org.xwiki.webjars.internal.FilesystemResourceReferenceSerializerTest.serializeCSSResourceWithURLsInIt whose code can be checked here. Now, it seems that this test case, and others in the same module, are dealing with actual files. The assertions seem to verify if some files exist. How are these files created? Could the outcome of another test influence this test case? When the code is mutated there may be some unexpected results reflected in the files created/erased/modified. Could this be the case? The methods being mutated are org.xwiki.webjars.internal.FilesystemResourceReferenceCopier.processCssfile and org.xwiki.webjars.internal.FilesystemResourceReferenceCopier.isRelativeURL

[surli]

@oscarlvp Hi we got another example of mutation score changing. This time on module xwiki-platform-observation-remote, the threshold is set at 80 and we got a build we reached only 75, see there

This time I saved the reports before triggering back the build. You can find them attached: build 160 was failing, 161 was ok. platform-observation-remote-pitreport-build-161.zip platform-observation-remote-pitreport-build-160.zip

[vmassol]

Note that today (2020-02-12) we had an unstability again with no code change AFAICS:

[ERROR] Failed to execute goal org.pitest:pitest-maven:1.4.10:mutationCoverage (pitest-check) on project xwiki-commons-extension-repository-maven: Mutation score of 85 is below threshold of 88 -> [Help 1]

I updated the mutation score to the current value at https://github.com/xwiki/xwiki-commons/commit/ad888d72791aade28c6b0b45b3d89d71af779027#diff-4e06773273323f2703b4c3a54f9afd47R36 and it passed on the CI yesterday and the days before and today, suddenly, it failed in https://ci.xwiki.org/job/xwiki-commons_pitest/724/console

[vmassol]

And another one found today: `[ERROR] Failed to execute goal org.pitest:pitest-maven:1.4.10:mutationCoverage (pitest-check) on project xwiki-platform-mailsender: Mutation score of 26 is below threshold of 44 -> [Help 1]``

I tried it locally and I have different mutation scores everytime I run it. Sometimes 9%, sometimes 34%, sometimes 18%, etc.

In the logs I see plenty of the following:

1:46:24 PM PIT >> INFO : Created  5 mutation test units
stderr  : Exception in thread "smtp:127.0.0.1:3025" java.lang.IllegalStateException: Can not open server socket for smtp:127.0.0.1:3025
        at com.icegreen.greenmail.server.AbstractServer.initServerSocket(AbstractServer.java:115)
        at com.icegreen.greenmail.server.Abstrstderr  : actServer.run(AbstractServer.java:86)
Caused by: java.net.BindException: Address already in use (Bind failed)
        at java.net.PlainSocketImpl.socketBind(Native Method)
        at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387)
        at java.net.Sestderr  : rverSocket.bind(ServerSocket.java:390)
        at java.net.ServerSocket.bind(ServerSocket.java:344)
        at com.icegreen.greenmail.server.AbstractServer.openServerSocket(AbstractServer.java:71)
        at com.icegreen.greenmail.server.AbstractServer.initServerSocket(Abstracstderr  : tServer.java:110)
        ... 1 more
stderr  : Exception in thread "smtp:127.0.0.1:3025" java.lang.IllegalStateException: Can not open server socket for smtp:127.0.0.1:3025
        at com.icegreen.greenmail.server.AbstractServer.initServerSocket(AbstractServer.java:115)
        at com.icegreen.greenmail.server.Abstrstderr  : actServer.run(AbstractServer.java:86)
Caused by: java.net.BindException: Address already in use (Bind failed)
        at java.net.PlainSocketImpl.socketBind(Native Method)
        at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387)
        at java.net.Sestderr  : rverSocket.bind(ServerSocket.java:390)
        at java.net.ServerSocket.bind(ServerSocket.java:344)
        at com.icegreen.greenmail.server.AbstractServer.openServerSocket(AbstractServer.java:71)
        at com.icegreen.greenmail.server.AbstractServer.initServerSocket(Abstracstderr  : tServer.java:110)
        ... 1 more

However this happens only when pitest/descartes executes.

Note: I've tried to check the generated mutations but it doesn't work. When using the EXPORT feature, I get an error:

[INFO] Defaulting target classes to match packages in build directory
5:46:19 PM PIT >> INFO : Verbose logging is disabled. If you encounter a problem, please enable it before reporting an issue.
5:46:20 PM PIT >> INFO : Sending 8 test classes to minion
5:46:20 PM PIT >> INFO : Sent tests to minion
5:46:20 PM PIT >> INFO : MINION : 5:46:20 PM PIT >> INFO : Checking environment

5:46:20 PM PIT >> INFO : MINION : 5:46:20 PM PIT >> INFO : Found  14 tests

5:46:20 PM PIT >> INFO : MINION : 5:46:20 PM PIT >> INFO : Dependency analysis reduced number of potential tests by 0

5:46:20 PM PIT >> INFO : MINION : 5:46:20 PM PIT >> INFO : 14 tests received
                                                                                                                                                                                                         -5:46:23 PM PIT >> INFO : Calculated coverage in 3 seconds.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  27.643 s
[INFO] Finished at: 2020-02-12T17:46:23+01:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.pitest:pitest-maven:1.4.10:mutationCoverage (pitest-check) on project xwiki-platform-mailsender: Execution pitest-check of goal org.pitest:pitest-maven:1.4.10:mutationCoverage failed: Type javax/mail/Multipart not present: javax.mail.Multipart -> [Help 1]
[ERROR] 

I've also made sure that the mail SMTP socket is closed after each test. Didn't help. FTR here's what I did in MailSenderApiTest:

    @AfterAll
    public static void afterAll() throws Exception
    {
        mailserver.stop();
        while(!isTcpPortAvailable(3025)) {
            Thread.sleep(100L);
        }
    }

    private static boolean isTcpPortAvailable(int port) {
        try (ServerSocket serverSocket = new ServerSocket()) {
            // setReuseAddress(false) is required only on OSX,
            // otherwise the code will not work correctly on that platform
            serverSocket.setReuseAddress(false);
            serverSocket.bind(new InetSocketAddress(InetAddress.getByName("localhost"), port), 1);
            return true;
        } catch (Exception ex) {
            return false;
        }
    }

Any idea is most welcome.

[vmassol]

Another one: [ERROR] Failed to execute goal org.pitest:pitest-maven:1.4.10:mutationCoverage (pitest-check) on project xwiki-platform-vfs-api: Mutation score of 62 is below threshold of 67 -> [Help 1]

[oscarlvp]

@vmassol the case of the xwiki-platform-mailsender is definitively an interference between the mutants. Remember that a mutant may corrupt the state of the test. If the code of the test has global effects, as it seems to be the case with this one, then it is very difficult to ensure the integrity of the system for the execution of the next mutant. Also , waiting for the port to be ready may make the test fail, remember that PIT always sets a timeout. On my side I will try to reproduce the issues with xwiki-commons-extension-repository-maven and xwiki-platform-vfs-api.

[oscarlvp]

Executing the following goal on xwiki-commons-extension-repository-maven

mvn clean install -Pquality -Dxwiki.pitest.skip=false -Djacoco.skip=true

ends in the following error:

[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.xwiki.extension.repository.xwiki.internal.XWikiExtensionRepositoryTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.32 s - in org.xwiki.extension.repository.xwiki.internal.XWikiExtensionRepositoryTest
[INFO] Running org.xwiki.extension.repository.xwiki.internal.SystemHTTPProxyTest
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.054 s <<< FAILURE! - in org.xwiki.extension.repository.xwiki.internal.SystemHTTPProxyTest
[ERROR] testProxy  Time elapsed: 1.054 s  <<< ERROR!
org.xwiki.component.manager.ComponentLookupException: Failed to lookup component [org.xwiki.extension.repository.xwiki.internal.XWikiExtensionRepositoryFactory] identified by type [interface org.xwiki.extension.repository.ExtensionRepositoryFactory] and hint [xwiki]
	at org.xwiki.component.embed.EmbeddableComponentManager.getInstance(EmbeddableComponentManager.java:204)
	at org.xwiki.test.mockito.MockitoComponentMockingRule.getComponentUnderTest(MockitoComponentMockingRule.java:243)
	at org.xwiki.extension.repository.xwiki.internal.SystemHTTPProxyTest.testProxy(SystemHTTPProxyTest.java:56)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:567)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.xwiki.test.mockito.MockitoComponentMockingRule$1.evaluate(MockitoComponentMockingRule.java:188)
	at com.github.tomakehurst.wiremock.junit.WireMockRule$1.evaluate(WireMockRule.java:62)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
	at org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:43)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
	at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
	at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
	at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
	at org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:82)
	at org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:73)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:248)
	at org.junit.platform.launcher.core.DefaultLauncher.lambda$execute$5(DefaultLauncher.java:211)
	at org.junit.platform.launcher.core.DefaultLauncher.withInterceptedStreams(DefaultLauncher.java:226)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:199)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:132)
	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:150)
	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:124)
	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
Caused by: org.xwiki.component.phase.InitializationException: Failed to create JAXB context
	at org.xwiki.extension.repository.xwiki.internal.XWikiExtensionRepositoryFactory.initialize(XWikiExtensionRepositoryFactory.java:67)
	at org.xwiki.component.embed.InitializableLifecycleHandler.handle(InitializableLifecycleHandler.java:39)
	at org.xwiki.component.embed.EmbeddableComponentManager.createInstance(EmbeddableComponentManager.java:365)
	at org.xwiki.component.embed.EmbeddableComponentManager.getComponentInstance(EmbeddableComponentManager.java:451)
	at org.xwiki.component.embed.EmbeddableComponentManager.getInstance(EmbeddableComponentManager.java:201)
	... 50 more
Caused by: javax.xml.bind.JAXBException: Implementation of JAXB-API has not been found on module path or classpath.
 - with linked exception:
[java.lang.ClassNotFoundException: com.sun.xml.internal.bind.v2.ContextFactory]
	at javax.xml.bind.ContextFinder.newInstance(ContextFinder.java:131)
	at javax.xml.bind.ContextFinder.find(ContextFinder.java:318)
	at javax.xml.bind.JAXBContext.newInstance(JAXBContext.java:478)
	at javax.xml.bind.JAXBContext.newInstance(JAXBContext.java:435)
	at javax.xml.bind.JAXBContext.newInstance(JAXBContext.java:336)
	at org.xwiki.extension.repository.xwiki.internal.XWikiExtensionRepositoryFactory.initialize(XWikiExtensionRepositoryFactory.java:65)
	... 54 more
Caused by: java.lang.ClassNotFoundException: com.sun.xml.internal.bind.v2.ContextFactory
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:602)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
	at javax.xml.bind.ServiceLoaderUtil.nullSafeLoadClass(ServiceLoaderUtil.java:92)
	at javax.xml.bind.ServiceLoaderUtil.safeLoadClass(ServiceLoaderUtil.java:125)
	at javax.xml.bind.ContextFinder.newInstance(ContextFinder.java:128)
	... 59 more

[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Errors: 
[ERROR]   SystemHTTPProxyTest.testProxy:56 » ComponentLookup Failed to lookup component ...
[INFO] 
[ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 0

[vmassol]

@oscarlvp that's weird, it works fine on our CI + locally (just retested now). Maybe try with "-U" to make sure you have recent deps?

I ran it from xwiki-commons/xwiki-commons-core/xwiki-commons-extension/xwiki-commons-extension-repositories/xwiki-commons-extension-repository-maven.

What JDK are you using?

I'm on:

$ java -fullversion
openjdk full version "1.8.0_202-b08"

[oscarlvp]

I'm usind JDK 13 openjdk full version "13.0.1+9" so, that might be the problem right?

[vmassol]

so, that might be the problem right?

yes could well be. It's possible that they remove some stuff or that it's optional, etc. On our side we build with java 8.

[vmassol]

Got a new one (from https://ci.xwiki.org/job/xwiki-platform_pitest/558/console):

[ERROR] Failed to execute goal org.pitest:pitest-maven:1.4.10:mutationCoverage (pitest-check) on project xwiki-platform-observation-remote: Mutation score of 78 is below threshold of 80 -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.pitest:pitest-maven:1.4.10:mutationCoverage (pitest-check) on project xwiki-platform-observation-remote: Mutation score of 78 is below threshold of 80

Same as https://github.com/STAMP-project/pitest-descartes/issues/62#issuecomment-470542825

[vmassol]

Another flicker from today:

[ERROR] Failed to execute goal org.pitest:pitest-maven:1.4.10:mutationCoverage (pitest-check) on project xwiki-platform-user-default: Mutation score of 87 is below threshold of 88

[vmassol]

Another flicker today:

[ERROR] Failed to execute goal org.pitest:pitest-maven:1.4.10:mutationCoverage (pitest-check) on project xwiki-platform-webjars-api: Mutation score of 22 is below threshold of 24 -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.pitest:pitest-maven:1.4.10:mutationCoverage (pitest-check) on project xwiki-platform-webjars-api: Mutation score of 22 is below threshold of 24

Unless the following change could cause the mutation score to change but then I'd be curious to know why: https://github.com/xwiki/xwiki-platform/commit/9ebbeb6505454485df2382b2044b4a3f83b23f97

Thanks!