lettuce icon indicating copy to clipboard operation
lettuce copied to clipboard

--failfast: broken tests do not show up properly in xunit output

Open mhaas opened this issue 6 years ago • 0 comments

When a step fails when lettuce is running with --failfast, the generated xunit output is incorrect. It will contain the testcase with a <failure> child element, but the failures or errors attributes on the <testsuite> root element is not update.

Shortened example output:

<?xml version="1.0" encoding="UTF-8"?>
<testsuite errors="0" failures="0" hostname="localhost" name="lettuce" tests="268" time="0" timestamp="2018-03-23T14:31:54">
  <testcase classname="My cool feature" name="My cool step" time="2.592967"/>
  <testcase classname="My broken feature" name="My failing step" time="30.65113">
    <failure message="My failing step..."/>...</failure>
</testsuite>

Further output of the lettuce command also indicates that the failure is not propagated correctly:

2018-03-23T15:07:02.826107825Z  Lettuce aborted running any more tests because was called with the `--failfast` option
2018-03-23T15:07:02.826110123Z  
2018-03-23T15:07:02.826112230Z  9 features (9 passed)
2018-03-23T15:07:02.826114560Z  9 scenarios (9 passed)
2018-03-23T15:07:02.826116786Z  268 steps (268 passed)
2018-03-23T15:07:02.888016366Z  Traceback (most recent call last):
2018-03-23T15:07:02.888054476Z    File "/var/jenkins/curing_backend/virtualenv/env_1/bin/lettuce", line 11, in <module>
2018-03-23T15:07:02.888058167Z      sys.exit(main())
2018-03-23T15:07:02.888060666Z    File "/var/jenkins/curing_backend/virtualenv/env_1/lib/python2.7/site-packages/lettuce/bin.py", line 134, in main
2018-03-23T15:07:02.901923524Z      result = runner.run()
2018-03-23T15:07:02.901940441Z    File "/var/jenkins/curing_backend/virtualenv/env_1/lib/python2.7/site-packages/lettuce/__init__.py", line 218, in run
2018-03-23T15:07:02.907028817Z      raise LettuceRunnerError("Test failed.")
2018-03-23T15:07:02.907043373Z  lettuce.exceptions.LettuceRunnerError: Test failed.

mhaas avatar Mar 23 '18 15:03 mhaas