Test output as JUnit.xml.

[QGhd]

trafficstars

Dear Sebastian (@awvwgk),

I use your nice and neat FORTRAN test driver. Thank you for sharing it!

To integrate the FORTRAN tests into an Azure DevOps build pipeline (means: let the test results being displayed properly), I have added a functionality to write the test results to a JUnit.xml file that can be consumed by Azure DevOps. I referred to this xml schema: https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd. (This might be an alternative: https://github.com/junit-team/junit5/blob/22a64e7aad5ae0e3c9cf2a692f77bd3de8d1bbed/platform-tests/src/test/resources/jenkins-junit.xsd, but I have not used it.)

Are you interested to get my changes resp. to integrate them into your project? If yes, how should I submit my code to you?

Kind regards,

Oliver

[awvwgk]

Thanks for your interest in this project. Contributions are welcome here, if you have an implementation or the plan to implement it yourself.

Given the way test-drive collects and runs tests I can imagine that a different routine for collecting tests would already be sufficient.

[QGhd]

I have attached two changed source files (replacing src/testdrive.f90 and test/main.f90) as well as the resulting JUnit.xml file that is written along with the screen output. This xml is doing the job for us in an Azure DevOps pipeline based on the data that is also written on the screen. The main.f90 uses 4 additional external subroutines from testdrive.f90. Two to open and close the Junit.xml and two to open and close the xml tag. The rest of the xml output is covered internally in testdrive.f90. However, I assume that you would prefer to avoid these additional functions, but I choose a simple approach with minimal code changes, instead of a major rework of the module structure.

In addition, the JUnit.xml file references to an xml schema (JUnit.xsd from https://github.com/windyroad/JUnit-Schema). In theory the JUnit.xml should be conform to this schema, but it is not for several reasons:

1. There are missing (required) xml tag attributes, like number of all/failed/succeeded/skipped tests per testsuite that are as summarized data not available at the time, when the according opening xml tag is written.
2. There are missing xml tag attributes, like execution time, that are not stored in the code (as far as I could see).
3. There is no corresponding xml tag in the schema file for the additional stdout/stderr output per testcase. Therefore, I added according xml tags () per testcase that are - following the xsd - only allowed once per testsuite. Fortunately, Azure DevOps does not care about the discrepancy, so we are fine with this setup (but it is not a generic solution).

I have also attached another xml schema file (jenkins-junit.xsd from https://github.com/junit-team/junit5 resp. https://junit.org/junit5/). To be honest I am not sure, what should be regarded as the standard for a JUnit.xml. In fact, I do not care much, since Azure DevOps test pipeline works fine. Others, however, may have a more specific opinion on the subject.

Let me know, if you want to discuss my changes or whether I can support you. Thank you for considering my changes.

JUnit.zip

[awvwgk]

I think it is easier to open a pull request or post a patch here.

[QGhd]

Will do.

[QGhd]

Done: https://github.com/fortran-lang/test-drive/pull/27

[awvwgk]

Thanks a lot for your work on the JUnit support. I (finally) took some time to incorporate it with test-drive in https://github.com/fortran-lang/test-drive/pull/42.

Some important remarks, we cannot continuously write to an IO unit because we might produce an incomplete xml file if a test fails for whatever reason, therefore we need to cache the output in a string and periodically override the whole JUnit.xml output file. Also, this strategy allows to collect some metrics while we are processing the tests and creating the xml output, which we cannot do while we write to an IO unit.

Feedback on the implementation in https://github.com/fortran-lang/test-drive/pull/42 is very welcome.