Implement Maven plugin to check threshold based on pseudo-tested and partially tested methods

[oscarlvp]

Create a mechanism to fire a build failure in Maven when the mutation score drops below certain configurable value or the number of strong/weak pseudo tested methods raises above an also configurable threshold.

[vmassol]

Note that this issue was raised as a result of the workshop I did with Oscar on XWiki, see my blog post at http://massol.myxwiki.org/xwiki/bin/view/Blog/MutationTestingDescartes

[vmassol]

It would be great to have this issue implemented ASAP so that we can test it on the xwiki project. I believe it's quite easy to do and would allow putting PIT/Descartes in production and get feedback. Thanks!

[oscarlvp]

As we discussed briefly last time, this is already supported by PIT and can be used. You may want to check the following PIT configuration parameters:

• mutationThreshold: An integer specifying the mutation score threshold at which the build must fail.
• maxSurviving: An integer specifying the maximum number of surviving mutants that a project may allow.

Anyways, I would like to leave this issue open in order to inspect the possibility of implementing a PIT Mojo that fails on the number of pseudo-tested or partially tested methods found, which makes more sense in the context of descartes.

[vmassol]

ok thanks Oscar, I must have forgotten. The strong pseudo-tested criteria makes senses indeed.

BTW, I've tried to use PIT 1.3.1 this morning and I've rebuilt Descartes from sources (0.2-SNAPSHOT - would be nice to have some release BTW) and it failed (some classes/methods were not found). So there's some incompatibility between latest PIT and latest Descartes versions.

[oscarlvp]

Given the current PIT code, we might have to create a new Maven plugin to implement a task that fails on the number of pseudo-tested and partially-tested methods.

[oscarlvp]

We have a custom reporter in Descartes which is identified as METHODS. This reporter produces a JSON file with a list of pseudo-tested and partially-tested methods. This result can be used to implement the Maven plugin.

[oscarlvp]

Probably the steps are:

• Implement a new PITest Maven goal, see here. A good approach is to inherit from AbstractPitMojo, in the same way it is done in PitMP.
• This goal, should use Descartes as the mutation engine, and METHODS as the reporter. See here
• Inspect the report produced to fail the build if the numbers of pseudo-tested or partially-tested methods go beyond a given threshold.

[spookyvale]

About the last option, I have a plugin to visualize Descartes reports in Jenkins. You can find the initial version here. The plugin currently has an option to mark the build as UNSTABLE if the global Mutation Coverage is below a certain treshold (it was a proof of concept), but it can be improved to meet the goal.

[vmassol]

Note that the use case of Jenkins is different IMO.

For example in the XWiki case it's not interesting: We need to be able to do this at the level fo the maven build so that it can be executed on developer machines (and since it's in the build it can of course be executed on the CI too, and fail the build too ;)).

Note that we're currently using pitest maven plugin's check mojo.

[nicolabertazzo]

hi @vmassol , we are working on the development of maven plugin too. The jenkins plugin could be useful to display the trend of the metric collected from the mutation testing execution.

[vmassol]

I agree that the historical use case (trends) is interesting and not something you can do on the dev machines.

But you don't need the ability to fail the build in the Jenkins plugin for computing trends! ;) (my comment was a reply about the fail-the-CI-job feature).

Also note that it could be interesting to have a Sonar plugin for the trend.

[nicolabertazzo]

I created a fork of pit-test where I added partialyTested and pseudoTested Threshods: https://github.com/nicolabertazzo/pitest/tree/pseudo-partialy-test I try to add this features to pitmp but the method to extend (AbstractPitMojo.execute()) is marked as final.

[nicolabertazzo]

I added two new parameters to AbstractPitMojo of pitest project (i forked it):

• pseudoTestedThreshold
• [partiallyTestedThreshold] (https://github.com/nicolabertazzo/pitest/blob/c69f46ccb88b1395600b755fcc9fb72f81eed031/pitest-maven/src/main/java/org/pitest/maven/AbstractPitMojo.java#L346)

And check the thresolds in AbstractPitMojo.execute()

In this way I can compute the pseudo-tested or partially-tested methods for every execution on pitest (with any mutation engine).

The other solution that was suggested from Oscar was to analyze the report produced from decartes methods report engine (<outputFormat>METHODS</outputFormat>) This solution works only if you use pit with decartes as mutator engine and methods as report engine.

I tryed to add my solution directly to pitmp-maven-plugin project but I not able to extend AbstractPitMojo.execute() because it is marked as final.

Do you agree with my solution or do you prefer to analyze the report produced from decartes (with methods as report engine). Do you have any hint for integrate this solution to pitmp-maven-plugin project?

[vmassol]

Do you have any hint for integrate this solution to pitmp-maven-plugin project?

You already answered it I think since you said:

I try to add this features to pitmp but the method to extend (AbstractPitMojo.execute()) is marked as final.

So this just needs to not be final. I think it's a very bad practice in general to have final methods btw ;) (my personal POV)

[oscarlvp]

The concepts of pseudo-tested and partially-tested methods make sense only when using descartes, so there is no problem in using the reporter. The goal to be implemented in the new implementation of AbstractPitMojo should use only descartes as mutation engine.

[Cael35]

@nicolabertazzo I had the same issue developing PitMP, indeed many methods are private or final in AbstractPitMojo. So I overwrote shouldRun(), called and the beginning of execute(), and analyse() which actually does the work.

[nicolabertazzo]

Ok thanks for the hints.

[nicolabertazzo]

To implement Maven plugin to check threshold based on methods i tryed these 3 solutions:

1. Fork pit project (https://github.com/nicolabertazzo/pitest/commits/pseudo-partialy-test)

   • added two new parameter to pit and checked partialyTested and pseudoTested thresholds on AbstractPitMojo.java
   • store partialyTestedDetected and pseudoTestedDetected on MutationStatistics.java
   • compute partialyTestedDetected and pseudoTestedDetected on MutationStatisticsListener.java This solution worked but it's not formally corrected because it coupled pit with descartes

2. Extend Pitmp (https://github.com/STAMP-project/pitmp-maven-plugin)

   • I tried to extend execute() and analyse() methods of AbstractPitMojo but it's not enought because I need to compute pseudoTested and partialyTested methods (on MutationStatisticsListener) and get the result on analyse method (so extend MutationStatistics.java). I dont able to implement the solution in pitmp

3. Develop a new plugin that check the report created from pit-decartes mutator with METHODS as report engine (similar to the jenkins plugin developed from @spookyvale ). The main problem is that i didn't find a way to get the path of the last report produced from decartes. (the solution that i want to investigate is convert the output folder name as date and get the most recent date)

[oscarlvp]

@nicolabertazzo You can combine your options 2 and 3. You can implement a Maven Mojo inside PItMP that injects the METHODS output format. This format is a mutation listener in the same way MutationStatisticsListener is. Right now it only produces a JSON file, but we can modify or extend the class to contain properties you can use for the Mojo, or whatever suits your needs, even creating a new listener that makes the build fail. If you want to check the JSON file directly, PIT, by default, creates a folder with the timestamp. So, as you say you can go for the latest report according to the date, and even sorting the folders by name.

[vmassol]

@nicolabertazzo It would really better if you didn't have to fork pit. Have you opened a ticket on the PIt project to discuss it with them and to see what could be done to make PIT extensible for this use case/needs? This would be the best solution by far.

Note that XWiki is using the PIT Maven plugin and not PIT MP (since it takes way too long to execute on the xwiki codebase). So a solution based on PIT would be good.

[nicolabertazzo]

As suggested from @oscarlvp and @CaelInria i extended pitmp-maven-plugin project. I Created a pull request where:

• I added two parameters: pseudoTestedThreshold and partiallyTestedThreshold to pitmp mojo
• I added a MutationResultListenerFactory (CHECKTHRESHOLDS) that fail the build if the number of partialy/pseudo tested methods are above the thresholds

[oscarlvp]

This issue can be transferred to the PitMP repository.