JUnit fails to report results of some tests if used with surefire and parallel option including methods

[apierzch]

Example: run mvn test in JUnitParams project with following configuration:

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.18.1</version>

                <configuration>
                    <runOrder>filesystem</runOrder>
                    <parallel>classesAndMethods</parallel>
                    <perCoreThreadCount>false</perCoreThreadCount>
                    <threadCount>16</threadCount>
                    <useUnlimitedThreads>false</useUnlimitedThreads>
                    <threadCountSuites>0</threadCountSuites>
                    <threadCountClasses>0</threadCountClasses>
                    <threadCountMethods>0</threadCountMethods>
                    <parallelOptimized>true</parallelOptimized>
                    <includes>
                        <include>**/SubclassTest.java</include>
                        <include>**/Samples_*</include>
                    </includes>
                </configuration>
            </plugin>

This should give total of 44 tests as of today, but often it will give less than that. Contact me for details about research done so far. Suspicious places:

• org.apache.maven.surefire.junitcore.JUnitCoreRunListener#fillTestCountMap
• org.apache.maven.surefire.junitcore.ClassesParallelRunListener#checkIfTestSetCanBeReported
• org.apache.maven.surefire.junitcore.TestSet#setAllScheduled

Currently following values of surefire "parallel" option should not be used:

• "classesAndMethods"
• "methods"
• "both"