add JUnit XML as an output-format for report

[jclerman]

Certain CI tools (CircleCI, and apparently Hudson/Jenkins) have dedicated functionality to store and display test results, but they seem to have standardized on a format that isn't currently one of robot report's output options: JUnit XML. I'm considering writing a little script to convert one of the output formats into Junit XML as a postprocessing step, but it'd be great - and probably useful to others using robot to run tests in a CI environment - to have the output directly supported in robot.

Some relevant links:

• https://stackoverflow.com/questions/4922867/what-is-the-junit-xml-format-specification-that-hudson-supports
• https://circleci.com/docs/2.0/collect-test-data/

[jamesaoverton]

Thanks for raising this. I need a bit of help to understand the details.

We use JUnit and Maven Surefire already. When I run mvn verify I see XML files under robot-core/target/surefire-reports/. The Circle CI doc you link to says that they support Surefire and JUnit.

What is the missing piece?

[jclerman]

Hi @jamesaoverton. I am pretty sure you're referring to tests of the robot code, whereas I'm talking about tests run by robot itself, via the robot report command. The workflow I have in mind is one where I'm using robot in a CI workflow to verify that my ontology passes certain tests - and if/when those tests fail, I'd like to use the CI tools to see the details of the failures.

[jamesaoverton]

Ok, now I understand. Yes, I was talking about unit and integration tests of the ROBOT code base, but you mean that the if the output of robot report was in one of those supported XML formats then your CI system would display the problems more nicely. That makes good sense.

We output these formats for robot report: TSV, CSV, HTML, YAML, JSON, XLSX

https://github.com/ontodev/robot/blob/master/robot-core/src/main/java/org/obolibrary/robot/ReportOperation.java#L494

We could add JUnit-XML, but it would have to contain basically the same information. So in the end it wouldn't be much different from whatever TSV to XML script you probably have in mind.

What would you want the XML to look like?

[jclerman]

tbh, I'm just learning about this format myself. My use-case involves CircleCI, and their documentation of the format leaves something to be desired. However, these are relevant links:

• CircleCI test-summary troubleshooting
• https://github.com/levlaz/circleci-test-results-validator

More complete documentation here.

Based on the StackOverflow link above, here's a sample of what the XML might look like - but admittedly it's a bit challenging to map the robot report paradigm (many tests, each with an effectively-unknown number of "test cases"/possible failures) to this format:

<testsuite tests="3"> <!-- count of SPARQL-query tests-->
    <testcase classname="ERROR" name="file:ASuccessfulErrorTest.rq"/>
    <testcase classname="WARN" name="file:ASuccessfulWarnTest.rq"/>
    <testcase classname="ERROR" name="file:AFailingErrorTest.rq">
        <failure type="ERROR"> details about failure (subject/property/value triple?) </failure>
    </testcase>
</testsuite>

There's also a supported gatherer of test suites (you can have multiple suites), but it seems to me that the queries specified in robot report --profile more closely correspond to Junit's test-cases than to test-suites.

One key thing to note: The number of queries (test-cases), and the name of each, is included in this format, which is more than what's provided in the existing robot report formats, which only indicate failures, not a list of all tests run. The Report class would need some extension to include that list of run tests (queries).

[jamesaoverton]

Ok, that all makes sense. The HTML table format is probably the closest to what you want, but I can see that it might not include everything you want.

To be completely honest, I can't see any of my clients or team members using this feature in the near-to-medium term, so I can't justify spending resources to build it. But I would be happy to accept a PR, and give you some support, if you want to work on this and contribute it to ROBOT.

If you just end up with a script to convert one of the existing formats to XML, and you want to contribute that, then we could add it to the repo here somewhere.