Introduce a new key for test framework output displayed in HTML report

[maks-rafalko]

We are integrating Stryker's HTML report into @infection, and I would like to discuss a new key for displaying the whole test framework output for a particular Mutant

Current behavior

Currently, there is a statusReason key that is created to display only "assertion message", as I was told by @nicojs. However, in order to better understand why particular Mutant was killed or not, it would be nice to have the whole test framework output.

Suggested solution

Probably, it would be nice to add a new key like testFrameworkOutput / testRunnerOutput to store it.

Please see the current behavior:

and desired behavior:

As you can see on the first screenshot, output from statusReason is not formatted, because it is wrapped with <h6> instead of <pre> as on the 2nd screenshot.

On Infection Playground, we have the same feature to display the whole output which is IMO very useful, see the right of the image:

Alternative solution

Instead of adding a new key, statusReason can be used, but <h6> should be replace with <pre> to correctly format the message with \n

[nicojs]

I think we should call it testRunnerOutput. @hugo-vrijswijk @rouke-broersma, thoughts?

[rouke-broersma]

Seems fine to me. More important I think is finding a good way to display this. And do we make this a field of mutant, unit test or both?

[nicojs]

Yeah, good questions. I thought it was for mutants only. Probably @maks-rafalko will be able to answer in detail.

[maks-rafalko]

this should definitely be a field of mutant in schema, right near statusReason, because this is an output of tests, executed for particular Mutant .

Regarding displaying, it's up to you guys, but what I want to underline is that this output can be quite big, depending on the tests and test runner. For example, for one of the real Mutant and PHPUnit test framework, it can be like this (I've added <pre> tag manually here to display it correctly):

output as test:

PHPUnit 9.5.4 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.0.13
Configuration: /tmp/infection/phpunitConfiguration.afc645b550c8be9f496a9d89e85b3da6.infection.xml

Testing 
F

Time: 00:00.006, Memory: 10.00 MB

There was 1 failure:

1) Infection\Tests\Logger\Html\StrykerHtmlReportBuilderTest::test_it_logs_correctly_with_mutations with data set "different mutations" (Infection\Metrics\MetricsCalculator Object (...), Infection\Metrics\ResultsCollector Object (...), array('1', array(90, 50), array(array('php', '<?php\n\ndeclare(strict_types...}\n}\n', array(array('32f68ca331c9262cc97322271d88d06d', 'PublicVisibility', 'protected function add(int $a... : int', 'Replaces the `public` method ...cted`.', array(array(13, 5), array(13, 45)), 'Killed', 'PHPUnit output. Tests: 1, Ass...ons: 3', array('06a6c58caae5aa33e9b787f064618f5e'), array(), 1), array('fd66aff56e903645c21271264b062b4f', 'MethodCallRemoval', '', 'Removes the method call.', array(array(15, 9), array(15, 27)), 'Survived', 'PHPUnit output. Tests: 1, Ass...Failed', array('06a6c58caae5aa33e9b787f064618f5e'), array('06a6c58caae5aa33e9b787f064618f5e'), 1), array('746519c01522ddc7da799a9b7927e4c2', 'MethodCallRemoval', '', 'Removes the method call.', array(array(17, 9), array(19, 11)), 'Survived', 'PHPUnit output. Tests: 1, Ass...set #1', array('2b67abde50b026f4057311ea32409632'), array('2b67abde50b026f4057311ea32409632'), 1), array('633b144fb6d55bbc60430df68a952388', 'ArrayItemRemoval', '$this->innerArray(array_keys(...2']));', 'Removes an element of an arra...ion.\n', array(array(28, 9), array(28, 65)), 'Survived', 'PHPUnit output. Tests: 3, Ass...ons: 3', array('06a6c58caae5aa33e9b787f064618f5e', '949bee6dd4ac608462995babbe81ee12', '2733f8c97b5ba92b1aacb77d46837b0e'), array(), 3))), array('php', '<?php\n\ndeclare(strict_types...}\n}\n', array(array('12f68ca331c9262cc97322271d88d06d', 'PublicVisibility', 'protected function add(int $a... : int', 'Replaces the `public` method ...cted`.', array(array(13, 5), array(13, 6)), 'Killed', 'Output without ability to det... tests', array('06a6c58caae5aa33e9b787f064618f5e'), array(), 0)))), array(array(array(array('06a6c58caae5aa33e9b787f064618f5e', 'TestClass::test_method1'), array('2b67abde50b026f4057311ea32409632', 'TestClass::test_method1 with ...set #1'))), array(array(array('949bee6dd4ac608462995babbe81ee12', 'TestClass2::test_method2'), array('2733f8c97b5ba92b1aacb77d46837b0e', 'TestClass2::test_method3')))), array('Infection', array('https://infection.github.io/', 'https://infection.github.io/i...go.png'))))
Failed asserting that two arrays are identical.
--- Expected
+++ Actual
@@ @@
             )
         )
     )
-    'testFiles' => Array &37 (
-        '/infection/path/to/TestClass.php' => Array &38 (
-            'tests' => Array &39 (
-                0 => Array &40 (
-                    'id' => '06a6c58caae5aa33e9b787f064618f5e'
-                    'name' => 'TestClass::test_method1'
-                )
-                1 => Array &41 (
-                    'id' => '2b67abde50b026f4057311ea32409632'
-                    'name' => 'TestClass::test_method1 with data set #1'
-                )
-            )
-        )
-        '/infection/path/to/TestClass2.php' => Array &42 (
-            'tests' => Array &43 (
-                0 => Array &44 (
-                    'id' => '949bee6dd4ac608462995babbe81ee12'
-                    'name' => 'TestClass2::test_method2'
-                )
-                1 => Array &45 (
-                    'id' => '2733f8c97b5ba92b1aacb77d46837b0e'
-                    'name' => 'TestClass2::test_method3'
-                )
-            )
-        )
-    )
-    'framework' => Array &46 (
+    'testFiles' => Array &37 ()
+    'framework' => Array &38 (
         'name' => 'Infection'
-        'branding' => Array &47 (
+        'branding' => Array &39 (
             'homepageUrl' => 'https://infection.github.io/'
             'imageUrl' => 'https://infection.github.io/images/logo.png'
         )
     )
 )

/home/maksrafalko/apps/infection/tests/phpunit/Logger/Html/StrykerHtmlReportBuilderTest.php:79
/home/maksrafalko/apps/infection/vendor/phpunit/phpunit/phpunit:61

FAILURES!
Tests: 1, Assertions: 1, Failures: 1.

[maks-rafalko]

by the way, here is a full HTML report generated for Infection itself, probably you will find it helpful during implementation of this feature request (if you decide to add it). This HTML report contains test framework outputs in statusReason fields, and as you can see - it's not readable for the end user.

Unfortunately GitHub does not allow uploading html files, so I've added it to zip archive.

infection.zip

[nicojs]

That's very useful yes! I think the easiest solution might be to put it in the horizontal drawer, as it is designed for a lot of text.

[rouke-broersma]

It should at the very least not open my default.