build doesn't fail on cucumber test failures

[amazingankur]

Buildkite passes the build even though cucumber tests get failed. Any help is appreciated.

[moensch]

A build step fails if any of your commands return a non-zero exit code.

I assume cucumber is returning non-zero if there are failed tests? Are you by any chance wrapping your call to cucumber in some bash script that doesn't have set -e set and are running commands after the call to cucumber?

In general, if you are invoking bash scripts from a pipeline command step, you ought to set -e to make it bail out if any command fails. You'll see this at the top of most bash scripts within build pipelines:

#!/bin/bash
set -ueo pipefail

<stuff>

This means:

• Error if any bash variable is undefined (unbound)
• If any instruction in the script returns non-zero, exit the script
• If something to the left of a pipe returns non-zero, also fail

[amazingankur]

I am running simple sh commands like git clone/pull followed by mvn test In code i am running below: KarateStats stats = CucumberRunner.parallel(getClass(), 10, karateOutputPath); Above line provides the number of failures in cucumber. Not sure how this input can be read by build I am failrly new to build pipelines so excuse me for obvious/silly questions

[moensch]

I just wanted to highlight with my response that there isn't too much special about how Buildkite handles things. If you have a step like this for example:

steps:
 - label: run test
   command:
    - git clone some stuff
    - mvn test

If the mvn test command returns a non-zero exit code, then this step will fail and hence mark your build as a failure.

The important thing is that if you want a step to fail, you need to make sure a command in your step returns a non-zero exit code. So I'd start debugging this locally and try to get it to a point where your script that invokes the tests does indeed return a non-zero exit code when tests fail.

[amazingankur]

Thanks much! Above answer gives me a start. I'll just post once I am finished.

[amazingankur]

Working fine once i put Assert on test results for failed test cases. Thanks and appreciate the help.