Option to abort on first failure

[KittyGiraudel]

Current behavior:

All tests are run even if one fails.

Expected behavior:

Providing an option to abort the test suite as soon as a test fails.

How to reproduce the current behavior:

Make a test fail.

Environment

• Cypress Version: 0.19.2

[KittyGiraudel]

In the mean time, this does the trick:

Cypress.on('fail', () => Cypress.runner.abort())

[brian-mann]

In 0.20.0 this became:

Cypress.on('fail', () => {
  Cypress.stop()
})

[KittyGiraudel]

From what we’ve seen, Cypress.stop() exits the process with a 0 code, which means even if a test fails, this will make it behave like everything went fine. We believe Cypress.runner.abort() did exit with a non-0 code in case of a failure.

Are we mistaken?

[brian-mann]

I didn't try it, but Cypress.stop() does do more things than aborting just the runner.

Try this:

Cypress.runner.stop()

That's where the abort code moved to, but even this does a bit more than it used it. May work though.

[tomaswitek]

Any update here? Both Cypress.runner.stop() and Cypress.stop() behave like everything went fine.

[brian-mann]

Nope. PR's are welcome.

[ThomasDecavele]

This is a much desired feature!

In the meanwhile I managed to get the workaround working though, here's what's in my "support/index.js":

import './commands'
import { OperationCanceledException } from 'typescript';

Cypress.on('fail', () => {
    Cypress.stop();
    throw new OperationCanceledException();
});

[nanoflat]

I think it will work in support/index.js, not in plugins/index.js

[01taylop]

The use-case I have for this is that I need my authentication tests to pass first - if they don't all subsequent tests will fail. Although the other tests will fail in the before hooks, it still slows things down unnecessarily (30 seconds per test file).

I have to name my test 1auth so that it runs first and whitelist a cookie which contains a token required for the other tests to pass.

Cypress.Cookies.defaults({
  whitelist: 'token',
})

I was hoping to do something like this in my auth tests to stop running tests on fail;

  before(() => {
    Cypress.on('fail', () => {
      Cypress.runner.stop()
    })
  })

It does stop the test runner, but it doesn't fail any tests so I get 3/3 tests passed and a green build which is not the intended outcome. I also tried adding am intentional failing test before stopping cypress, but then the test runner continues.

[danh91]

I have a temporary solution that I think can help someone.

I just add an after each in my utilities file imported by all my specs.

afterEach(function() {
  if (this.currentTest.state === 'failed') {
    Cypress.runner.stop()
  }
});

I take advantage of Mocha offering a way to check the state of the last test. And the nice part is that the process exit with a failing code (not 0)

[sanderdejong88]

@DanH91 I've been following this thread for months, and finally someone found a way! Awesome! I've put your code in the support file which also seems to work.

[harvitronix]

This no longer works for me with Cypress 3.x.

[jennifer-shehane]

@harvitronix We changed how specs are run in 3.0 so that the files are run isolated. This means the before runs before each spec file - so I am guessing that it is stopping and failing that 1 spec file and then continuing to the next file. Is this what is happening?

[brian-mann]

Yes we've waited to implement this feature due to how aborting is much more complex now.

We likely need to allow for multiple abort strategies

abortStrategy: 'spec' | 'run'

One would abort just the spec, one would abort the entire run.

This becomes even more complex with parallelization.

[harvitronix]

@jennifer-shehane that's exactly right. Totally understand it gets much more complex now. Would love to have the options you describe, @brian-mann.

[vedmant]

Is there a way to stop execution but run after() method?

[mattvb91]

:+1:

[mohsenny]

I tried your solutions and the runner correctly stops. However, I'm looking for a way to only stop the current test scenario/test file/describe(), but let the runner keep running to execute the rest of scenarios/test files/describe()s. Any suggestion?

An example: As you can see, the 3rd test file (Device Type) failed and the runner correctly didn't execute the rest of the test cases in that file, but it also didn't go any further with the 4th test file (Device) either.

[drumheadrob]

Hi, loving Cypress, but not being able to stop our CI build for a failing test is stopping me from being able to use this in CI, and will especially be a problem in CD.

I've searched quite thoroughly and hope I'm missing something somewhere, but it seems like this is related to this thread?

If so, is there any progress / planned work on this solution? Cheers.

[bahmutov]

@drumheadrob your issue is very different - you are saying that the cypress run does NOT exit with code 1 after all tests complete and some fail?!

[jennifer-shehane]

We are currently working on the underlying processes needed to 'cancel' a run and any other queued spec files, which is a precursor to the work needed to 'abort' the rest of the tests on first failure.

[pacop]

Some news right here? I am facing the very same problem, we were not able to use cypress on our CI.

[stclairdaniel]

I'm also interested in this.

[nantoniazzi]

Are you still working on this feature ? The more we add tests with Cypress, the more it becomes unusable in our CI

[cmcnicholas]

We have the same situation, paying for cypress dashboard on private repos but our tests (now ~400 or so) are taking more and more time to complete. Much needed to stop early so our pipeline is freed up for the next PR to execute.

[vladarefiev]

It will be cool feature

[gurre]

In order to fail the test make sure to throw the error.

// cypress/support/index.js
Cypress.on('fail', error => {
  Cypress.runner.stop()
  throw error; // throw error to have test fail
});

[itslenny]

@gurre is this working for you?

I'm doing this in my main describe block. The test run is stopping, but it's exiting as if it was successful.

  before(() => {
    Cypress.on('fail', error => {
      Cypress.runner.stop()
      throw error;
    });
  });

edit: nvm... I totally missed @DanH91 's answer above ( https://github.com/cypress-io/cypress/issues/518#issuecomment-373369129 ) which works perfectly with no need to explicitly throw.

[TrueMan777]

I wanted to do the fail-fast-all type of approach so I've ended up adding the following to my cypress/support/index.js file. Hopefully, it helps someone else.

// Fail-fast-all-files
before(function () {
    cy.getCookie('has-failed-test').then(cookie => {
        if (cookie && typeof cookie === 'object' && cookie.value === 'true') {
            Cypress.runner.stop();
        }
    });
});

// Fail-fast-single-file
afterEach(function () {
    if (this.currentTest.state === 'failed') {
        cy.setCookie('has-failed-test', 'true');
        Cypress.runner.stop();
    }
});

[vesper8]

@TrueMan777

Hi and thanks for sharing your solution! I'm rather new to Cypress. I tried dumping the above code inside my cypress/support/index.js but that is throwing an unexpected error. Do you need to wrap it or do something extra?

This is my index.js before and after

// Import commands.js using ES2015 syntax:
import './commands';

// Import commands.js using ES2015 syntax:
import './commands';

// Fail-fast-all-files
before(function () {
    cy.getCookie('has-failed-test').then(cookie => {
        if (cookie && typeof cookie === 'object' && cookie.value === 'true') {
            Cypress.runner.stop();
        }
    });
});

// Fail-fast-single-file
afterEach(function () {
    if (this.currentTest.state === 'failed') {
        cy.setCookie('has-failed-test', 'true');
        Cypress.runner.stop();
    }
});

I am now receiving this error:

/node_modules/@babel/runtime-corejs2/helpers/esm/typeof.js:1
import _Symbol$iterator from "../../core-js/symbol/iterator";
^
ParseError: 'import' and 'export' may appear only with 'sourceType: module'

This occurred while Cypress was compiling and bundling your test code. This is usually caused by:

- A missing file or dependency
- A syntax error in the file or one of its dependencies

Fix the error in your code and re-run your tests.

Any idea what's wrong? Thanks!!