[SUREFIRE-2252] - Fix JUnit5 reporting when tests are executed in parallel

[ArvindJoshi-okta]

Problem

• When parallel tests are executed in JUnit5, the reporting (console and xml) have some issues
• The tests are executed but the reporting combines tests into other classes showing incorrect results (as seen below)
• Sample code is attached to the JIRA https://issues.apache.org/jira/browse/SUREFIRE-2252

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.example.ClassBIT
[INFO] Running org.example.ClassAIT
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.026 s – in org.example.ClassAIT
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.026 s – in org.example.ClassBIT
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0

Fix

• Issue was in JUnitPlatformProvider where the execute method was sending in all the classes at the same time in class selector list, hence combining the results
• Fix was to move the list creation inside the iteration loop
• Added relevant unit and integration tests

Checklist

Following this checklist to help us incorporate your contribution quickly and easily:

• [x] Make sure there is a JIRA issue filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes.
• [x] Each commit in the pull request should have a meaningful subject line and body.
• [x] Format the pull request title like [SUREFIRE-XXX] - Fixes bug in ApproximateQuantiles, where you replace SUREFIRE-XXX with the appropriate JIRA issue. Best practice is to use the JIRA issue title in the pull request title and in the first line of the commit message.
• [x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
• [x] Run mvn clean install to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
• [x] You have run the integration tests successfully (mvn -Prun-its clean install).

If your pull request is about ~20 lines of code you don't need to sign an Individual Contributor License Agreement if you are unsure please ask on the developers list.

To make clear that you license your contribution under the Apache License Version 2.0, January 2004 you have to acknowledge this by using the following check-box.

• [x] I hereby declare this contribution to be licenced under the Apache License Version 2.0, January 2004

• [ ] In any other case, please file an Apache Individual Contributor License Agreement.

[michael-o]

There are test failures, please take a look.

[ArvindJoshi-okta]

There are test failures, please take a look.

Looks like the POM issue I was referring to in the JIRA, on what value should I add to test my current logic/version.

org.apache.maven.plugin.PluginResolutionException: Plugin org.apache.maven.plugins:maven-surefire-plugin:3.3.2-SNAPSHOT or one of its dependencies could not be resolved: Could not find artifact org.apache.maven.plugins:maven-surefire-plugin:jar:3.3.2-SNAPSHOT

[ArvindJoshi-okta]

There are test failures, please take a look.

Looks like the POM issue I was referring to in the JIRA, on what value should I add to test my current logic/version.

org.apache.maven.plugin.PluginResolutionException: Plugin org.apache.maven.plugins:maven-surefire-plugin:3.3.2-SNAPSHOT or one of its dependencies could not be resolved: Could not find artifact org.apache.maven.plugins:maven-surefire-plugin:jar:3.3.2-SNAPSHOT

Will try to use <version>${surefire.version}</version> in my test pom.xml and see if that gets picked up automatically.

[ArvindJoshi-okta]

There are test failures, please take a look.

@michael-o Could you pls review now? All tests are passing.

[henjovr]

How close is this to being merged?

It is incredibly useful to us..

[michael-o]

How close is this to being merged?

It is incredibly useful to us..

Merging is simple with a button, but understanding that this is right is hard. For everything else there is Mastercard.

[ArvindJoshi-okta]

I am going to close this PR since its not the right approach

[jira-importer]

Resolve #2625