lettuce
lettuce copied to clipboard
--failfast: broken tests do not show up properly in xunit output
When a step fails when lettuce is running with --failfast
, the generated xunit output is incorrect. It will contain the testcase with a <failure>
child element, but the failures
or errors
attributes on the <testsuite>
root element is not update.
Shortened example output:
<?xml version="1.0" encoding="UTF-8"?>
<testsuite errors="0" failures="0" hostname="localhost" name="lettuce" tests="268" time="0" timestamp="2018-03-23T14:31:54">
<testcase classname="My cool feature" name="My cool step" time="2.592967"/>
<testcase classname="My broken feature" name="My failing step" time="30.65113">
<failure message="My failing step..."/>...</failure>
</testsuite>
Further output of the lettuce command also indicates that the failure is not propagated correctly:
2018-03-23T15:07:02.826107825Z Lettuce aborted running any more tests because was called with the `--failfast` option
2018-03-23T15:07:02.826110123Z
2018-03-23T15:07:02.826112230Z 9 features (9 passed)
2018-03-23T15:07:02.826114560Z 9 scenarios (9 passed)
2018-03-23T15:07:02.826116786Z 268 steps (268 passed)
2018-03-23T15:07:02.888016366Z Traceback (most recent call last):
2018-03-23T15:07:02.888054476Z File "/var/jenkins/curing_backend/virtualenv/env_1/bin/lettuce", line 11, in <module>
2018-03-23T15:07:02.888058167Z sys.exit(main())
2018-03-23T15:07:02.888060666Z File "/var/jenkins/curing_backend/virtualenv/env_1/lib/python2.7/site-packages/lettuce/bin.py", line 134, in main
2018-03-23T15:07:02.901923524Z result = runner.run()
2018-03-23T15:07:02.901940441Z File "/var/jenkins/curing_backend/virtualenv/env_1/lib/python2.7/site-packages/lettuce/__init__.py", line 218, in run
2018-03-23T15:07:02.907028817Z raise LettuceRunnerError("Test failed.")
2018-03-23T15:07:02.907043373Z lettuce.exceptions.LettuceRunnerError: Test failed.