Produce JUnit-compatible test report XML

[magnusbaeck]

Logstash Filter Verifier should be able to produce a JUnit-compatible test report XML file so that test results can be machine processed and e.g. be displayed in CI systems like Jenkins.

In order to do this the test result handling should be refactored to be more pluggable. Right now it's hardcoded to only do one thing (echo everything to stdout) and we shouldn't add test report XML creation in the same manner; both actions should be two implementations of the same interface.

[disaster37]

+1 I need also this feature for my CI.

[filex]

Maybe a shortcut to Jenkins could be a TAP output. That seems to be quite close to the structure of the current output and is still somewhat human readable, and there are Jenkins plugins for TAP, of course :)

The LFV 1.2.1 output is:

Running tests in test.json...
Comparing message 1 of 5 from test.json (Test 1)...
Comparing message 2 of 5 from test.json (Test 2)...
Comparing message 3 of 5 from test.json (Test 3)...
Comparing message 4 of 5 from test.json (Test something else)...
--- /tmp/298483256/test.json/1/expected	2017-05-19 18:21:26.341970050 +0000
+++ /tmp/298483256/test.json/1/actual	2017-05-19 18:21:26.341970050 +0000
@@ -1,6 +1,6 @@
 {
   "@timestamp": "2017-05-18T16:55:29.000Z",
-  "agent": "a",
+  "agent": "ua",
   "auth": "-",
   "bytes": 218828,
   "clientip": "172.16.200.X",
Comparing message 5 of 5 from test.json...

A TAP output could look like this:

1..5
ok 1 - Test 1 (test.json)
ok 2 - Test 2 (test.json)
ok 3 - Test 3 (test.json)
not ok - Test something else (test.json)
ok 5 - message 5 (test.json)

Testcase 1 to 4 use the description field. Case 5 has the default description.

However, as the diff of the failed test cannot be printed into the TAP output, we would need some sort of switch for that.

[Amorik]

Is outputting a report in JUnit/NUnit/other a possibility with the changes since this was last discussed? If so, is there any guidance how to accomplish that?