libtest
libtest copied to clipboard
Added very basic JUnit output
This PR adds a very basic JUnit formatter, and should probably not be merged without further discussion.
This output format is widely used, but sadly is not standardized. The best summary I could find is was in Catch's website.
Some talk points:
- Some data and attributes are currently stubbed, such as the test-suite and class names.
- It is assumed only one test suite exists. (I'm not sure this concept is relevant to libtest's tests).
- Tests are given the run time "0" as they are currently not measured by the framework (benchmarks are given real time).
- It seems as if std-out/err are owned by the test-suite as far as JUnit is concerned, which is not the case in libtest. Should we concat the output of the [Failing?] testcases?
- We should probably give command-line flags to set the test-suite name, or somehow infer it from cargo. (or both)
- The chrono crate is used to format the timestamp in ISO8601, but if the dependency is unwanted, I can always rewrite this one function.
- Ideally I would want to test this feature either by using the junitparser python package or by validating the output using a JUnit crate, but I'm not sure if that's the best solution. Maybe I should just compare it to a known good output.
Thoughts?
Just wanted to say that I'm excited about JUnit format potentially being added!
FWIW, In my experience the JUnit result files emitted by cargo-junit seem to be "correct", at least in the sense that various CI engines are able to correctly parse/understand those JUnit files (albeit with missing timestamps). Sharing in case that's of any use as a comparison point
Thank you! I'll make sure to compare those
I'm curious as to why the timings for the tests aren't recorded? Most test frameworks I've worked with reports duration of tests OOTB, and I don't see a good reason for libtest to not do so.
The main reason is that Nobody has sent a PR implementing that.
+1 on this! I'd love to see this PR merged, especially since we have --report-time which seems sufficient to report test times and TrResult captures stdout (maybe expandable to stderr as well?). I was gearing up to write yet another own cargo subcommand (as cargo-junit seems to just hang with my nightly, and cargo-suity doesn't handled ignored on the crates.io published version), but this is a much cleaner way to go, and would go far in integrating w/ Jenkins and other established CI tooling. Any way I can help this PR move through?
Testing the output against some CI framework will be nice, but I think that this is repo is semi-dead for some reason. (no commits or merges since april)
We should probably find the maintainer somewhere else.
Edit: In another PR it is mentioned that this repo is actually not the crate that is distributed with Rust. I have no idea why this isn't mentioned in the README
@iostat It's probably needed to retarget this PR against the rustc codebase, but unfortunately I no longer have leisure to do so. If you could do it, I think it would be great.
+1 on this as well. I'm eagerly awaiting a built in exporter to junit rather than an ad-hoc one.
Hey guys, any chance we can get this effort going again? I think it'd be quite convenient to have this in.