[SUREFIRE-2041] Ordering test classes and methods according to -Dtest property

[winglam]

trafficstars

This PR is a new implementation for PR#348 and PR#495. ​ The changes in this pull request enable Surefire to run test classes and test methods in any order specified by a newly added <runOrder> (testorder).

PR#348 attempted to implement this feature but @Tibor17 suggested that such a feature should be implemented differently. PR#495 supports only test class ordering, while this PR supports test class ordering and test method ordering within each test class.

Overall, the changes include​

• A new <runOrder> called testorder that would make Surefire run test classes and test methods in the order specified by <test>. Regex and include/exclude are supported as is the case for other runOrders
• Tests for the newly added features ​

The newly added testorder <runOrder> is supported for JUnit 3 and JUnit 4 tests. We have also implemented support for JUnit 5, but we are unsure on how we should get and sort all the test methods before sorting them for JUnit 5. Unlike JUnit 4, JUnit5 does not provide a interface like org.junit.runner.Request#sortWith() in JUnit4 to help us sort the methods. To help with this issue, we current rely on Java reflection to get the methods and then sort them. For this reason, we created another PR to show that change, but we left the changes out of this PR. Please let us know if you have any suggestions for supporting JUnit 5. ​ Simple (no regex) example using Http-Request:

mvn test -Dsurefire.runOrder=testorder \
-Dtest=com.github.kevinsawicki.http.HttpRequestTest#getUrlEncodedWithPercent,\
com.github.kevinsawicki.http.HttpRequestTest#uploadProgressSendReader,\
com.github.kevinsawicki.http.HttpRequestTest#malformedStringUrlCause,\
com.github.kevinsawicki.http.EncodeTest#encode

​ Output from Maven should say that the test class HttpRequestTest ran before EncodeTest

Tests run: 3 … in
com.github.kevinsawicki.http.HttpRequestTest
Tests run: 1 … in 
com.github.kevinsawicki.http.EncodeTest

​ The file HttpRequestTest.xml (http-request/lib/target/surefire-reports/TEST-com.github.kevinsawicki.http.HttpRequestTest.xml) should say

<testcase name="getUrlEncodedWithPercent"...>
<testcase name="uploadProgressSendReader"...>
<testcase name="malformedStringUrlCause...>

​ Regex example using Http-Request:

mvn test -Dsurefire.runOrder=testorder \
-Dtest=com.github.kevinsawicki.http.EncodeTest*,\
com.github.kevinsawicki.http.HttpRequestTest#getUrl*

​ Output from Maven should say

Tests run: 2 … in 
com.github.kevinsawicki.http.EncodeTest
Tests run: 4 … in
com.github.kevinsawicki.http.HttpRequestTest

​ The file HttpRequestTest.xml (http-request/lib/target/surefire-reports/TEST-com.github.kevinsawicki.http.HttpRequestTest.xml) should say

<testcase name=”getUrlEncodedWithPercent”...>
<testcase name="getUrlEncodedWithSpace"...>
<testcase name="getUrlEncodedWithUnicode”...>
<testcase name="getUrlEmpty”...>

​ Include/Exclude example using Http-Request:

mvn test -Dsurefire.runOrder=testorder \
-Dtest=com.github.kevinsawicki.http.HttpRequestTest#getUrl*,\
\!com.github.kevinsawicki.http.HttpRequestTest#getUrlEmpty

​ Output from Maven should say

Tests run: 3 … in com.github.kevinsawicki.http.HttpRequestTest

​ The file HttpRequestTest.xml (http-request/lib/target/surefire-reports/TEST-com.github.kevinsawicki.http.HttpRequestTest.xml) should say

<testcase name=”getUrlEncodedWithPercent”...>
<testcase name="getUrlEncodedWithSpace"...>
<testcase name="getUrlEncodedWithUnicode”...>