freshen
freshen copied to clipboard
rlisagor
Undefined steps does not fail a test
When I created a scenario and had no steps, it gave me the UNDEFINED error in the output...but all the tests were flagged as passed and nose displayed OK at the end.
Also a 0 return code was returned from the process instead of 1 (or some error code) to indicate failure.
This means it's hard for automated builds to know if the tests actually failed or not, since a test that was not run due to undefined steps does not fail the test run.
if that is the default behavior, freshen should offer an option to override it.
Agreed, an option to make it fail if there are undefined steps could be useful. Cucumber allows this with the --strict flag. Adding to the 1.0 milestone.
One hack to use this in CI/build system would be to pipe the output (or tee the output) to a file, then this:
grep UNDEFINED nose.log && (echo "ERROR: Undefined found"; exit 1)
That is exactly what we had to do. I wrote a Maven plugin to trigger Freshen runs for our Java apps and we had to add custom logic like that: https://github.com/jacek99/maven-python-mojos/blob/master/maven-python-mojos/maven-bdd-plugin/src/main/java/com/github/mojo/bdd/NoseMojo.java
Ah yes- this would work fine with a Jenkins job or in a build/deploy script. With bash, you could probably simplify it to:
grep UNDEFINED nose.log && exit 1
On 5 January 2012 14:55, Jacek Furmankiewicz [email protected] wrote:
That is exactly what we had to do. I wrote a Maven plugin to trigger Freshen runs for our Java apps and we had to add custom logic like that: https://github.com/jacek99/maven-python-mojos/blob/master/maven-python-mojos/maven-bdd-plugin/src/main/java/com/github/mojo/bdd/NoseMojo.java
Reply to this email directly or view it on GitHub: https://github.com/rlisagor/freshen/issues/16#issuecomment-3370706
Danny Staple
Director, ODM Solutions Ltd w: http://www.odmsolutions.co.uk Blog: http://orionrobots.co.uk/blog1-Danny-Staple
See https://github.com/abeld/freshen/commit/3bca046a89e2415eddf4ecc04953a5ba43507f62 for an implementation of the "add an option to turn undefined steps into failures" feature. (The branch is https://github.com/abeld/freshen/tree/undefined-steps-are-errors--option)
