Add flag to always return success status even when there are failed tests

[liyaka]

When running tests as a part of the CI process, it's critical that the test step will always return a success status so that the process can continue and just present tests results report. At the moment gotestsum returns non-zero status and causes our build to fail. On commandline it looks like this:

➜  blink git:(xxx) ✗ gotestsum -- ./pkg/errors/...
✖  pkg/errors (445ms)

=== Failed
=== FAIL: pkg/errors TestIsDuplicateKeyError/not_duplicate_key_error_1 (0.00s)
    errors_test.go:127: IsSQLError() = false, want true
    --- FAIL: TestIsDuplicateKeyError/not_duplicate_key_error_1 (0.00s)

=== FAIL: pkg/errors TestIsDuplicateKeyError (0.00s)

DONE 18 tests, 2 failures in 1.079s
➜  blink git:(xxx) ✗ echo $?
1

At the moment I have no choice but add || true at the end of the command which is obviously bad practice - if there is a real failure (not test failure), I won't be able to catch it. Please add a flag that will cause gotestsum to ignore test fails, I didn't find one in the documentation..

[dnephin]

Thank you for opening this issue! gotestsum is used extensively in CI, so it should be possible to make this work.

The exit code from gotestsum is pretty important, because bugs in test2json can sometimes cause misleading results in the output. If we can find an option that doesn't requiring masking the exit code, I think that would be better than adding a new flag.

Which CI system are you using? Most of the ones I'm familiar with provide some way to run a step even after previous steps have failed.

• GitHub actions has both step.if and job.if. You can use them with if: always() to have a step or a job always run, even after a failure in a previous step
• CircleCI has when which works the same way: when: always
• Jenkins seems to have a couple options for this as well: https://issues.jenkins.io/browse/JENKINS-43027

Does the CI system you are using have something similar?

Another option is to script it in bash. Something like this should work:

set +e
gotestsum ...
exit_code=$?
set -e

# present test report here

exit $exit_code

That should allow you to do more work after the tests fail.

[liyaka]

Using set +e in bash is the same as adding || true at the end of the command - you might miss the real crushes which are not test failures

[dnephin]

It's not the same. Using set +e allows you to capture the exit code in a variable $?, so that you can exit with that code later.