Catch2

Catch2 copied to clipboard

Description For the non-streaming, file based reporters (e.g. JUnit) a fatal crash of the test runner should still produce a meaningful report.

Meaningful in the sense, that any test case which had started execution should be marked as a fail even if Catch2 had encountered a fatal error, and every not started test should be marked as "not executed".

It is not possible to capture the execution time, or any test output of the crashing test case, but that's usually a non-issue.

The the current state, any (hard) crash results in no report file being written for the corresponding test suite at all. That even includes not writing any failure reports for detected test case failures prior to a hard crash.

The assumption that any crash can be "handled" by just intercepting the signal is wrong, in any scenario with heap corruptions etc. it is just plain dangerous to "handle" in any form. So hard crashes are actually still a common occurrence.

In technical terms: Provide the option to write out incremental reports immediately prior to the start of each test case execution.

Additional context In CI integration attempt this yields a blind spot with Catch2, as any test suite which doesn't manage to reach the final write-out will effectively never even register with any of the globing based JUnit aggregators.

A hard crash, or a test timeout which results in termination by external watchdog, is a common scenario.

The most important information is which test case of a test suite has caused a crash, be it as spurious as it only can be. Running test cases as individual processes is not a feasible default choice. Loosing the stdout and/or captured context is less of an issue, as a crash caused by a memory access violation or alike has to be analyzed with additional tool support anyway.

The additional IO overhead from replacing the result file over and over again is still significantly lower than reloading the test executable with a potentially huge dynamic dependency tree and cached state.

The same feature had previously proven valuable in a proprietary fork of GTest in bigger projects, so the behavior description above is well tested with regard to the interoperability in CI integrations.

Appears to be more tricky to implement with Catch2, as IStream is lacking any notion of streams which support seek/truncate semantics. But without truncating an already written (incremental) report this can't be implemented.

Patching the individual reporters to predict write out a predicted failure from IEventListener::testCasePartialStarting would be straightforward, otherwise.