mocha
mocha copied to clipboard
mochajs
xUnit xml format uses errors not failures
Hello,
We are using the built-in xUnit reporter as part of Mocha and are running into an issue where the xml generated treats assertion failures as errors. Specifically, when a test fails as a result of an assertion, the errors attribute on the
/* Actual output /
<testsuite name="Mocha Tests" tests="4" failures="0" errors="1" skipped="0" timestamp="Tue, 20 Oct 2020 15:24:27 GMT" time="136.099">
/ Expected output */
<testsuite name="Mocha Tests" tests="4" failures="1" errors="0" skipped="0" timestamp="Tue, 20 Oct 2020 15:24:27 GMT" time="136.099">
Ultimately for us, this causes the test report functionality of Bitbucket Pipelines to fail rendering the xml output. Comparing Jnit xml output, it uses the failures attribute for assertion failures rather than errors, which seems correct to me.
My expectation would be that errors are used for test execution problems, not test assertion failures and seems like it should be a bug. I have included the xml output below.
<testsuite name="Mocha Tests" tests="1" failures="0" errors="1" skipped="0" timestamp="Tue, 20 Oct 2020 15:24:27 GMT" time="136.099"> <testcase classname="Class Tests" name="should test the class" time="13.213"> <failure>Setting an on purpose fail so I can see if xunit results are being reported. AssertionError [ERR_ASSERTION]: Setting an on purpose fail so I can see if xunit results are being reported. at Context.<anonymous> (test/sample/sample.test.ts:76:16) at processTicksAndRejections (internal/process/task_queues.js:97:5)</failure> </testcase> </testsuite>
What is the expectation of errors vs failures for the xml file? We can work around the issue by post processing the xml file, but wanted to see if I should report a bug on this issue, or if it's working as expected.
Thanks,
Kevin
there was an issue about this before... I am not sure if there was a PR for it or what, but obviously it hasn't been addressed.
I think the main problem is that Mocha doesn't really know the difference between a "failure" and an "error". Why? Because all test failures come in the form of a thrown exception (or rejected Promise). They are errors. Thus, failures: 0 is hardcoded in the xunit reporter.
If an "error" in XUnit parlance means "test execution problem", we need to define "test execution problem". We could assume that any exception thrown outside of the context of a test is an "error" (e.g., in a hook, in Mocha itself, etc.), but this does not always hold--some people make assertions in hooks. Is misusing Mocha's API an "error"? For instance, calling done twice. What of uncaught exceptions or unhandled rejections?
We can (usually) recognize errors from Mocha itself, because have a code prop which matches /^ERR_MOCHA_/.
Further complicating things is that none of the current Mocha maintainers actually use the XUnit reporter, and we could really use help from somebody who does, and is interested in maintaining it.
To move this forward, I think what we need next is a definition of clear boundaries between "failure" and "error" in the context of Mocha. I can't do that myself, but I can certainly provide some direction on an eventual implementation.
As a current work-around, we've got a custom reporter that essentially treats all errors as failures. This works for us currently, but we would like to jump in and contribute to the XUnit reporter if we can. It looks like there are several links on the main README.md that will be useful for us to get started...is there anything else we should read before jumping in on this? Thanks again for your help!
Kevin