Failed unit tests leads to gulp error

[coldacid]

Running the unitTest target from the generated gulp scripts results in the following error when any of the test cases fail:

[17:26:20] 'unitTest' errored after 22 s
[17:26:20] Error: 1
    at formatError (C:\0-DATA\bamboo\xml-data\build-dir\NCAM-DEV-JOB1\Frontend\node_modules\gulp\bin\gulp.js:169:10)
    at Gulp.<anonymous> (C:\0-DATA\bamboo\xml-data\build-dir\NCAM-DEV-JOB1\Frontend\node_modules\gulp\bin\gulp.js:195:15)
    at Gulp.emit (events.js:107:17)
    at Gulp.Orchestrator._emitTaskDone (C:\0-DATA\bamboo\xml-data\build-dir\NCAM-DEV-JOB1\Frontend\node_modules\gulp\node_modules\orchestrator\index.js:264:8)
    at C:\0-DATA\bamboo\xml-data\build-dir\NCAM-DEV-JOB1\Frontend\node_modules\gulp\node_modules\orchestrator\index.js:275:23
    at finish (C:\0-DATA\bamboo\xml-data\build-dir\NCAM-DEV-JOB1\Frontend\node_modules\gulp\node_modules\orchestrator\lib\runTask.js:21:8)
    at cb (C:\0-DATA\bamboo\xml-data\build-dir\NCAM-DEV-JOB1\Frontend\node_modules\gulp\node_modules\orchestrator\lib\runTask.js:29:3)
    at removeAllListeners (C:\0-DATA\bamboo\xml-data\build-dir\NCAM-DEV-JOB1\Frontend\node_modules\karma\lib\server.js:325:7)
    at Server.<anonymous> (C:\0-DATA\bamboo\xml-data\build-dir\NCAM-DEV-JOB1\Frontend\node_modules\karma\lib\server.js:336:9)
    at Server.g (events.js:199:16)
    at Server.emit (events.js:129:20)
    at net.js:1419:10
    at process._tickCallback (node.js:355:11)

Because gulp returns a non-zero result code to the system, our CI platform treats this as a fatal build error and does not continue the build process or grab artifacts that were already generated earlier in the process.

This occurs with all versions of gulp since (and including) 3.8.7 and all versions of karma since 0.13.3.

[coldacid]

Seems that Karma will return an error code of 1 in a number of different situations, from failed tests to problems connecting to browsers for testing. I've fixed the issue locally by including gulp-util and using the following callback function inside the unitTest task to avoid gulp failing:

function karmaDone(exitCode) {
  if (exitCode !== 0) {
    gutil.log('Karma exited with code ' + gutil.colors.red(String(exitCode)));
  }
  done();
}

[dustinspecker]

Thank you for catching this.

I agree that Gulp shouldn't spew out a stack trace for a Karma error.

Just trying to understand: when Karma fails you're wanting the process to still return an exit code of 0? I don't know your use case, but I'd imagine wanting CI to halt on non-zero exit codes would be ideal in most cases.

To prevent Gulp spitting out a stack trace, the following can be used:

gulp.task('unitTest', ['lint', 'karmaFiles'], function (done) {
  var server = new $.karma.Server(karmaConf, function (errorCode) {
    if (errorCode !== 0) {
      console.log('Karma exited with error code ' + errorCode);
      done();
      return process.exit(errorCode);
    }
    done();
  });
  server.start();
});

This will still have the process return the actual exit code from Karma and not have a stack trace.

Currently, I'm inclined to add the snippet described here as a fix for the stack trace, but am interested in hearing the benefits of returning a 0 error code instead.

[coldacid]

The problem with non-zero returns is that with the way that Bamboo works, I can't force it to process the test results and fail then if any tests fail. I can make the test result parsing happen as an always-runs step, but then I can't have failed tests stop further steps in the build process -- always-run steps are executed at the very end.

If Karma returned different exit codes for tests failed or actual internal errors, then we could let Gulp complete if tests failed, and still cause an early stop when it or Karma actually fails. That kind of makes this a bug for them, too.

[coldacid]

Another option, by the way, would be to add a --force option flag similar to the --stage prod option, and have that decide whether or not to continue on a non-zero exit from Karma.