Report should display exceeded slowness threshold for each test

[johnkary]

Individual tests may set slowness thresholds independent of the configured test-suite threshold.

/**
 * This test's runtime would normally be under the suite's threshold, but
 * this annotation sets a lower threshold, causing it to be considered slow
 * and reported on in the test output.
 *
 * @slowThreshold 5
 */
public function testCanSetLowerSlowThreshold()
{
    $this->extendTime(10);
    $this->assertTrue(true);
}

But when these per-test slowness thresholds are exceeded, the report displays only the configured test-suite threshold is printed. This makes the report appear incorrect at first glance.

.SI.........                                                      12 / 12 (100%)

You should really fix these slow tests (>500ms)...
 1. 805ms to run JohnKary\PHPUnit\Listener\Tests\SomeSlowTest:testWithDataProvider with data set "Rock"
 2. 705ms to run JohnKary\PHPUnit\Listener\Tests\SomeSlowTest:testWithDataProvider with data set "Chalk"
 3. 601ms to run JohnKary\PHPUnit\Listener\Tests\SomeSlowTest:testWithDataProvider with data set "Jayhawk"
 4. 503ms to run JohnKary\PHPUnit\Listener\Tests\SomeSlowTest:testAnotherSlowTests
 5. 501ms to run JohnKary\PHPUnit\Listener\Tests\SomeSlowTest:testLongEndToEndTest
 6. 13ms to run JohnKary\PHPUnit\Listener\Tests\SomeSlowTest:testCanSetLowerSlowThreshold

In the above example, the default slowness threshold 500ms is set. But the last test runs in 13ms, which is not greater than the reported 500ms. This can be confusing.

Possible Solution 1

Remove the slowness thresholds from the report output. This removes the inaccuracy and the report simply contains a list of slow tests and their execution time. It remains easy to see which tests would benefit from investigating why they are slow.

Possible Solution 2

In addition to Solution 1, each slow test could output its slowness threshold along with execution time. For example:

.SI.........                                                      12 / 12 (100%)

You should really fix these slow tests...
 1. 805ms > 500ms running JohnKary\PHPUnit\Listener\Tests\SomeSlowTest:testWithDataProvider with data set "Rock"
 2. 705ms > 500ms running JohnKary\PHPUnit\Listener\Tests\SomeSlowTest:testWithDataProvider with data set "Chalk"
 3. 601ms > 500ms running JohnKary\PHPUnit\Listener\Tests\SomeSlowTest:testWithDataProvider with data set "Jayhawk"
 4. 503ms > 500ms running JohnKary\PHPUnit\Listener\Tests\SomeSlowTest:testAnotherSlowTests
 5. 501ms > 500ms running JohnKary\PHPUnit\Listener\Tests\SomeSlowTest:testLongEndToEndTest
 6. 13ms > 5ms running JohnKary\PHPUnit\Listener\Tests\SomeSlowTest:testCanSetLowerSlowThreshold

Any other ideas for improving the report output format?

[LasseRafn]

I think solution 2 is pretty good and preferred. Easily readable without losing information 👍

Wouldn't hate solution 1 though