libtest icon indicating copy to clipboard operation
libtest copied to clipboard

Added very basic JUnit output

Open Gilnaa opened this issue 6 years ago • 8 comments

This PR adds a very basic JUnit formatter, and should probably not be merged without further discussion.

This output format is widely used, but sadly is not standardized. The best summary I could find is was in Catch's website.

Some talk points:

  • Some data and attributes are currently stubbed, such as the test-suite and class names.
  • It is assumed only one test suite exists. (I'm not sure this concept is relevant to libtest's tests).
  • Tests are given the run time "0" as they are currently not measured by the framework (benchmarks are given real time).
  • It seems as if std-out/err are owned by the test-suite as far as JUnit is concerned, which is not the case in libtest. Should we concat the output of the [Failing?] testcases?
  • We should probably give command-line flags to set the test-suite name, or somehow infer it from cargo. (or both)
  • The chrono crate is used to format the timestamp in ISO8601, but if the dependency is unwanted, I can always rewrite this one function.
  • Ideally I would want to test this feature either by using the junitparser python package or by validating the output using a JUnit crate, but I'm not sure if that's the best solution. Maybe I should just compare it to a known good output.

Thoughts?

Gilnaa avatar Apr 25 '19 13:04 Gilnaa

Just wanted to say that I'm excited about JUnit format potentially being added!

FWIW, In my experience the JUnit result files emitted by cargo-junit seem to be "correct", at least in the sense that various CI engines are able to correctly parse/understand those JUnit files (albeit with missing timestamps). Sharing in case that's of any use as a comparison point

calebcartwright avatar May 08 '19 04:05 calebcartwright

Thank you! I'll make sure to compare those

Gilnaa avatar May 08 '19 04:05 Gilnaa

I'm curious as to why the timings for the tests aren't recorded? Most test frameworks I've worked with reports duration of tests OOTB, and I don't see a good reason for libtest to not do so.

Alxandr avatar Jul 26 '19 23:07 Alxandr

The main reason is that Nobody has sent a PR implementing that.

gnzlbg avatar Jul 27 '19 00:07 gnzlbg

+1 on this! I'd love to see this PR merged, especially since we have --report-time which seems sufficient to report test times and TrResult captures stdout (maybe expandable to stderr as well?). I was gearing up to write yet another own cargo subcommand (as cargo-junit seems to just hang with my nightly, and cargo-suity doesn't handled ignored on the crates.io published version), but this is a much cleaner way to go, and would go far in integrating w/ Jenkins and other established CI tooling. Any way I can help this PR move through?

iostat avatar Jan 25 '20 18:01 iostat

Testing the output against some CI framework will be nice, but I think that this is repo is semi-dead for some reason. (no commits or merges since april)

We should probably find the maintainer somewhere else.

Edit: In another PR it is mentioned that this repo is actually not the crate that is distributed with Rust. I have no idea why this isn't mentioned in the README

@iostat It's probably needed to retarget this PR against the rustc codebase, but unfortunately I no longer have leisure to do so. If you could do it, I think it would be great.

Gilnaa avatar Jan 25 '20 20:01 Gilnaa

+1 on this as well. I'm eagerly awaiting a built in exporter to junit rather than an ad-hoc one.

Progdrasil avatar Oct 16 '20 01:10 Progdrasil

Hey guys, any chance we can get this effort going again? I think it'd be quite convenient to have this in.

svenstaro avatar Apr 30 '21 00:04 svenstaro