Remove support for junit-xml, use Jinja2 instead

[seberm]

This PR removes support for the deprecated junit-xml library and uses the jinja2 to generate junit XML instead.

It also prepares support for various junit flavors (--flavor) and adds the possibility to specify user-defined junit jinja templates using the --template-path option. These options are mutually exclusive.

TODOs:

• [ ] Possibility to specify template variables directly from cmdline as key-values (e.g. --var <key> <value>) for a custom flavor / user-defined template?

Pull Request Checklist

• [ ] implement the feature
• [ ] write the documentation
• [ ] extend the test coverage
• [ ] update the specification
• [ ] adjust plugin docstring
• [ ] modify the json schema
• [ ] mention the version
• [ ] include a release note

[seberm]

Hello @happz , thanks for your early hints. It's still very early for a review. Many code parts will probably change, especially due to the Polarion report plugin.

[LecrisUT]

Is it possible to validate the junit.xml files created?

[seberm]

Is it possible to validate the junit.xml files created?

Currently, this functionality is not supported, and the XML files are not validated against any XSD schema, even when using junit-xml.

From python-junit-xml README.rst:

As there is no definitive Jenkins JUnit XSD that I could find, the XML documents created by this module support a schema based on Google searches and the Jenkins JUnit XML reader source code. File a bug if something doesn't work like you expect it to. For Bamboo situation is the same.

The junit-xml repository currently includes only Python unit tests, while TMT tests have very basic XML checks (polarion, default junit). However, there is no existing code that validates the entire XML file against an XSD schema.

I plan to create simple XSD schemas for each JUnit flavor supported by TMT and add tests validating the generated JUnit files against these schemas.

Additionally, it would be a good idea to have TMT dynamically validate the schema of the generated JUnit files. If a file does not conform to its respective schema, the report plugin should fail, unless e.g. the --force argument is used, which would allow it to continue despite the schema mismatch.

[happz]

I plan to create simple XSD schemas for each JUnit flavor supported by TMT and add tests validating the generated JUnit files against these schemas.

Additionally, it would be a good idea to have TMT dynamically validate the schema of the generated JUnit files. If a file does not conform to its respective schema, the report plugin should fail, unless e.g. the --force argument is used, which would allow it to continue despite the schema mismatch.

+1. Check out https://github.com/teemtee/tmt/pull/3153, it does similar validation when consuming results.yaml, but it's not a blocker, only a warning sign, at least for now. That said, these files are not created by tmt - junit files would be. Failing this particular validation would mean the plugin is creating a broken output, which would be a bug. I'm not sure --force would be needed - if users wanted to overcome this bug temporarily, they should be able to modify either the schema or the template, and --force would sort of normalize the behavior in my eyes.

[seberm]

/packit test

[seberm]

/packit test

[seberm]

I've moved the decorator which adds the properties to tmt.Result into the junit module (related PR https://github.com/teemtee/tmt/pull/3166). Hopefully, it's better now :).

[seberm]

Could someone help me fix the tests? I'm having trouble understanding why they are failing and whether the errors are related to my pull request. Is it possible that the python3-lxml package is not installed in some environments?

[seberm]

Could someone help me fix the tests? I'm having trouble understanding why they are failing and whether the errors are related to my pull request. Is it possible that the python3-lxml package is not installed in some environments?

Error: 
 Problem 1: conflicting requests
  - nothing provides python3.9dist(lxml) >= 4.9 needed by tmt+report-junit-1.36.dev888-1.20240828170910228891.pr3166.26.g07024055.el9.noarch from @commandline
  - nothing provides python3.9dist(types-lxml) needed by tmt+report-junit-1.36.dev888-1.20240828170910228891.pr3166.26.g07024055.el9.noarch from @commandline
 Problem 2: package tmt+all-1.36.dev888-1.20240828170910228891.pr3166.26.g07024055.el9.noarch from @commandline requires tmt+report-junit = 1.36.dev888-1.20240828170910228891.pr3166.26.g07024055.el9, but none of the providers can be installed

[LecrisUT]

python-lxml doesn't seem to be available on el9. Maybe you'll have to package it to epel9 or if there are alternatives... But the suspicious part is typed-lxml, why is that requested?

[seberm]

As discussed today, I've tried to fix the lxml dependency and converted the time="<float>" attributes to float using a jinja filter. Also, the float times are now enforced by default.xsd schema.

Please, feel free to test the output against the "old" version generated by junit-xml and against the systems where you think the junit.xml is currently used by users (e.g. import into reportportal).

Thanks for the hints!

[martinhoyer]

time is probably not in float?

FAILED test_report_junit.py::TestStateMapping::test_pass - AssertionError: Attribute mismatch at #document.testsuites.testsuite: found time="0.0"
assert '0' == '0.0'

[seberm]

time is probably not in float?

FAILED test_report_junit.py::TestStateMapping::test_pass - AssertionError: Attribute mismatch at #document.testsuites.testsuite: found time="0.0"
assert '0' == '0.0'

It was int in the testsuite:

<testsuite disabled="0" errors="0" failures="0" name="name" skipped="0" tests="1" time="0">

But it was a float in the testsuites tag :) :

<testsuites disabled="0" errors="0" failures="0" tests="1" time="0.0">

I am perfectly fine with a time as an integer. But I think the time types should be the same across all these tags.