robotframework-oxygen icon indicating copy to clipboard operation
robotframework-oxygen copied to clipboard

CICD is not failing when listener is failing

Open Tattoo opened this issue 2 years ago • 0 comments

The CICD currently relies on Robot Framework exit code to know if the test execution failed or not. However, in cases where the Oxygen listener fails, but the test execution otherwise passes, RF will return 0 and the CICD passes.

For example:

==============================================================================
Atest                                                                 | PASS |
10 critical tests, 10 passed, 0 failed
10 tests total, 10 passed, 0 failed
==============================================================================
[ ERROR ] Calling method 'output_file' of listener 'oxygen.listener' failed: MismatchArgumentException: parse_results expects 3 arguments but got 1
Output:  /[...]/output.xml
Log:     /[...]/log.html
Report:  /[...]/report.html

[python38][robotframework==3.2] Exited with 0

Since Oxygen is, in part, a listener, we need CICD to detect that listener has failed during the test execution

Tattoo avatar Sep 09 '21 11:09 Tattoo