Possible Bug: Test Strength at 100% Despite Zero Line and Mutation Coverage

[CaioCesar-cloud]

Hi team,

I'm currently configuring my test coverage analysis using targetClasses. At this stage, I haven't written any tests yet, so both line coverage and mutation coverage are showing as 0%, as expected.

However, I noticed that the test strength is being reported as 100%, which seems incorrect given that no tests have been executed.

Context: lineCoverage: 0%

mutationCoverage: 0%

testStrength: 100%

No tests have been implemented yet.

targetClasses is set correctly.

Is this a known issue or a potential bug? Could this be related to how test strength is calculated when there are no mutations or tests present?

Thanks in advance!

[microness]

Hi,

I’ve reviewed the behavior and the relevant code in MutationStatistics.class.

Problem

The reason you’re seeing testStrength as 100% even when no tests have been run is due to how PIT calculates test strength:

The MutationStatistics.class appears to be the class responsible for generating the console report in pitest-entry:

MutationStatistics.class

And here is that class, focus on that methods.

• void report(PrintStream out)
• int getTestStrength() & getPercentage(long total, long actual)

// ...
public void report(PrintStream out) {
        long var10001 = this.getTotalMutations();
        out.println(">> Generated " + var10001 + " mutations Killed " + this.getTotalDetectedMutations() + " (" + this.getPercentageDetected() + "%)");
        var10001 = this.getTotalMutationsWithoutCoverage();
        // focus on `getTestStrength()`
        out.println(">> Mutations with no coverage " + var10001 + ". Test strength " + this.getTestStrength() + "%");
        var10001 = this.numberOfTestsRun;
        out.println(">> Ran " + var10001 + " tests (" + this.getTestsPerMutation() + " tests per mutation)");
        out.println("Enhanced functionality available at https://www.arcmutate.com/");
}

// and here
public int getTestStrength() {
    return PercentageCalculator.getPercentage(
        this.getTotalMutationsWithCoverage(),
        this.getTotalDetectedMutations()
    );
}

PercentageCalculator.class - related class

PercentageCalculator.getPercentage(total, actual) is implemented as follows:

public static int getPercentage(long total, long actual) {
    if (total == 0L) return 100; // Here
    else if (actual == 0L) return 0;
    else return Math.round((float) actual * 100 / total);
}

• total = number of mutations with coverage
• actual = number of detected (killed) mutations

So when there are no mutations with coverage (total == 0), the method returns 100%.
This is why testStrength is 100% even though:

• No tests have been executed
• Line coverage is 0%
• Mutation coverage is 0%

The report() method then prints this value along with the number of mutations without coverage:

out.println(">> Mutations with no coverage " + this.getTotalMutationsWithoutCoverage()
            + ". Test strength " + this.getTestStrength() + "%");

Summary

In short, since there is no test code to evaluate the test strength, it seems that PIT assumes the tests are "strong" by default(?).

Of course, one could also argue that if there are no tests, the test strength should be 0%.

Once tests are added and mutations are covered, testStrength will be correctly calculated based on the actual detected mutations. seeing 100% in a project with no tests is expected given the current implementation.

Any feedback is welcome—I'm sharing this based on my own exploration, as I wanted to understand it before potentially contributing to the project.

Environment:

• pitest-maven: 1.21.0
• pitest-junit5-plugin: 1.2.3

Q. Does the pitest-maven plugin use pitest-entry? From what I can tell in the Maven plugin repository, the pitest-maven plugin includes pitest-entry.(Compile Dependencies below)