[jest-circus] missing fail() method

[Darep]

This might be a known issue, but I could not find an existing issue so creating one here 😊 Also, I guess fail() was a bit of an undocumented feature, but we rely on it in our app for some nice developer experience improvements.

💥 Regression Report

After upgrading to Jest v27 (with jest-circus as default) the fail() method is no longer defined.

ReferenceError: fail is not defined

Last working version

Worked up to version: 26.6.3

Stopped working in version: 27.0.0

Can circumvent in 27.x with testRunner: "jest-jasmine2" in jest.config.js

To Reproduce

Steps to reproduce the behavior:

1. Open a JS project with jest >= 27.0.0
2. Write a test that includes a fail() method call
3. Notice that any tests with a call to fail() might pass (depending on the structure), and you will see a "fail is not defined" error message in Jest v27 with jest-circus (works correctly with jest-jasmine2)

Expected behavior

Expected fail() to work by default, as before, without any changes to jest.config.js.

Link to repl or repo (highly encouraged)

See this repo for example of the regression: https://github.com/Darep/jest-circus-fail-method

Check the branch jasmine where the testRunner is changed and the tests run correctly 🙂

The repo also hilights the way we use fail(), just to give some background info & motivation from our use-case 😄

Run npx envinfo --preset jest

Paste the results here:

$ npx envinfo --preset jest
npx: installed 1 in 0.976s

  System:
    OS: macOS 11.4
    CPU: (12) x64 Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
  Binaries:
    Node: 14.16.1 - ~/n/bin/node
    Yarn: 1.21.1 - ~/n/bin/yarn
    npm: 6.14.12 - ~/n/bin/npm
  npmPackages:
    jest: ^27.0.6 => 27.0.6 

[m-gagnon]

Any update on this? Many of my integration tests are missing the correct messaging now that this is undefined, and its causing a lot of confusion.

[ChrisKatsaras]

Same here! Would love to have this issue alleviated sooner than later :)

[srmagura]

As a result of this issue, there is currently a discrepancy between @types/jest, which does define fail, and jest-circus, which does not define fail. Discussion in DefinitelyTyped

As a temporary workaround, you can define your own fail function:

function fail(reason = "fail was called in a test.") {
  throw new Error(reason);
}

global.fail = fail;

[eranelbaz]

Same here

[bitsmakerde]

Same here, too. No change after fix

[vala84no]

As a result of this issue, there is currently a discrepancy between @types/jest, which does define fail, and jest-circus, which does not define fail. Discussion in DefinitelyTyped

As a temporary workaround, you can define your own fail function:

function fail(reason = "fail was called in a test.") {
  throw new Error(reason);
}

global.fail = fail;

Unfortunately that's not equivalent. The problem I'm having is that I need to fail a test from a location where any throw will be caught. Is there any more equivalent option available?

[dereekb]

I just ran into a test where I was getting "fail() is undefined" and had assumed all this time that fail worked like it used to since it exists in @types/jest.

Just to clarify why this functionality is important:

it('should fail if the document does not exist.', async () => {
      await c.accessor.delete();

      const exists = await c.accessor.exists();
      expect(exists).toBe(false);

      try {
        await c.accessor.update(c.dataForUpdate());  // should not succeed
        fail();   // previously this would signal to Jest to properly end the test as a failure, but not get caught in the below.
      } catch (e) {
        expect(e).toBeDefined(); // expect an error occured
      }
});

The above code with Jest 28 will now incorrectly always succeed, as fail() throw an exception that gets caught by the catch.

We don't want to catch any error either though, as unexpected errors should result in a test failure rather than success. We just want the tests to succeed when failures are expect.

I went ahead and created some test utility functions so I can continue using this pattern. I did end up finding and resolving a few more bugs too. Now the example test looks like:

    import { itShouldFail, expectFail } from '@dereekb/util/test';
    
    ...

    itShouldFail('if the document does not exist.', async () => {
      await c.accessor.delete();

      const exists = await c.accessor.exists();
      expect(exists).toBe(false);

      await expectFail(() => c.accessor.update(c.dataForUpdate()));
    });

It will be published on npm with @dereekb/util@^8.1.0.

I'm not too familiar with the inner workings of Jest or why it dropped the previous functionality of fail(), but I imagine it could be brought back for the cases where we are looking for a specific error. Jest's it functionality could be extended with a function that looks for failures, (I.E. it.fail, or something) and then watch for a specific exception to be thrown that is thrown by failSuccessfully() or something to that manner. Someone more familiar with building Jest extensions may see a better way to implement it as an extension as well.

[bpossolo]

also running into this while trying to upgrade from jest 26 to jest 27..

ReferenceError: fail is not defined

[abdulbasit1149]

Any update on this?

[albertodiazdorado]

Any update on this?

Or at least some information as to:

• Why this feature has been removed
• How can we achieve what we used to achieve with fail

[niklass08]

Same here, still getting the fail is not defined error, any update or insight would be great

[joeskeen]

Seeing as this thread isn't moving towards an upcoming resolution in the jest-circus runner, I figured out how to restore the missing fail() functionality without re-implementing it. Essentially, if you install jest-jasmine2 and modify your Jest config to set "test-runner": "jest-jasmine2", you can now use fail() in your tests.

See https://stackoverflow.com/a/73922010/1396477

[atz3n]

Hi, just wanted to share the workaround I'm using. I have created a fail function using expect and a failing comparison. It also displays messages in an okayish way. It's not the cleanest solution, but it solves the problem mentioned here. Maybe it is helpful for someone.

function fail(message = "") {
    let failMessage = "";
    failMessage += "\n";
    failMessage += "FAIL FUNCTION TRIGGERED\n";
    failMessage += "The fail function has been triggered";
    failMessage += message ? " with message:" : "";

    expect(message).toEqual(failMessage);
}

[rayniel95]

Using the answer proposed here I tested if the same behavior could be applied to Jest. My theory was correct. It is possible to fail a Jest test if you call the done callback function with some param. Here an example:

    test('this test must fail', (done) => {
        done('hey Rainyel this is a failed test')
    });

Output:

● this test must fail

Failed: "hey Rainyel this is a failed test"

   6 |
   7 | test('this test must fail', (done) => {
>  8 |   done('hey Rainyel this is a failed test')
     |   ^
   9 |
  10 | });
  11 |

  at Object.<anonymous> (src/__tests__/test.super.ts:8:3)

[chadjaros]

This is still happening in 29.5.0

[vtgn]

Hi! How this incredibly stupid bug could still be present after a so long time?! I've never seen a test framework without a fail() method in my whole life. This not fixed regression after 2 years is a huge embarrassment for jest and a huge lack of serious.

[dandv]

@vtgn: maybe because there are bigger and older issues with Jest, like #6695.

[thehale]

Building off @rayniel95's answer...

Tests will fail with a timeout if done is never called.

test('passing done fails after timeout if unused', (done) => {
  // This test will fail after a few seconds
  // because `done` is never invoked.
})

Here's how to use done as a stand-in for fail:

// FUNCTION TO TEST
function invokeCallbackIf(condition: boolean, callback: () => void) {
  if (condition) callback();
}

// TESTS
test('callback invoked when true', () => {
  invokeCallbackIf(true, () => expect("test passes").toBeTruthy());
})

test('callback NOT invoked when false', (done) => {
  invokeCallbackIf(false, () => done("should not have been called"));
  done(); // <-- REQUIRED or the test will fail with a timeout
})

Note A properly functioning fail is still preferred over any done workaround:

1. fail(reason) has clearer semantics than done(reason)
2. fail avoids the "boilerplate" of passing it as a param and calling it without arguments.

[thehale]

I just saw @atz3n's workaround and decided to simplify it...

Write a custom fail function

function fail(message: string = '') {
  expect(`[FAIL] ${message}`.trim()).toBeFalsy();
}

Full example

// FUNCTION TO TEST
function invokeCallbackIf(condition: boolean, callback: () => void) {
  if (condition) callback();
}

// TESTS
function fail(message: string = '') {
  expect(`[FAIL] ${message}`.trim()).toBeFalsy();
}

test('callback NOT invoked when false', () => {
  invokeCallbackIf(true, () => fail("callback erroneously invoked"));
});

test('callback invoked when true', () => {
  invokeCallbackIf(true, () => expect('test passes').toBeTruthy());
});

---

Note This workaround is still inferior to the original fail

1. The original fail was automatically globally available.
2. In failure messages, code snippets showed the invocation of the original fail. This workaround shows the expect call in its implementation.

[vala84no]

I just saw @atz3n's workaround and decided to simplify it...

Write a custom fail function

function fail(message: string = '') {
  expect(`[FAIL] ${message}`.trim()).toBeFalsy();
}

Full example Note This workaround is still inferior to the original fail

1. The original fail was automatically globally available.
2. In failure messages, code snippets showed the invocation of the original fail. This workaround shows the expect call in its implementation.

There are already simple work-arounds like this in the thread above. There's also another way you've missed that it's inferior to the original fail: it simply doesn't work if it's executed within a callback where a parent performs a catch. E.g.:

function underTest(callback) {
  try {
    return callback();
  } catch (error) {
    return false;
  }
}

In cases like this any fail() call in callback will be silently ignored and the test still passes because all it does it throw an exception anyone can catch rather than actually mark the test as failed. That's a pretty big drawback.

[thehale]

There are already simple work-arounds like this in the thread above. There's also another way you've missed that it's inferior to the original fail: it simply doesn't work if it's executed within a callback where a parent performs a catch.

I saw the "workaround" that simply raises an error. Since, like you, I consider "errors" in a test a completely separate result from "failures" in a test, such a workaround is unsatisfactory for me, even if it weren't further plagued by incompatibility with try/catch blocks. Thus, my deliberate omission of any work to offer improvements on those suggestions.

I had thought my one-line function which you quoted does work in try/catch blocks, but upon further testing I'm realizing that jest's expect clauses simply raise a JestAssertionError, so it does suffer from the same problem :/

function invokeWithTry(callback) {
  try {
    return callback();
  } catch (err) {
    return err;
  }
}

function fail(message = "") {
  expect(`[FAIL] ${message}`.trim()).toBeFalsy();
}

test("fail works in a try/catch", () => {
  const result = invokeWithTry(() => fail("expected failure")); // Erroneously passes
  expect(result).toBeUndefined(); // Actually fails here b/c `result` is a `JestAssertionError`
});

This limitation also applies to expect().fail(message) from jest-extended since that matcher also raises an error instead of quietly recording a test failure.

[thehale]

After spending a few more hours on the problem this morning, I found a satisfactory solution using a custom matcher which builds off of the fail() matcher from jest-extended.

Use a Custom Matcher (works in try/catch blocks)

function toFail(_, message) {
  this.dontThrow();  // <-- Enables recording failures in try/catch blocks
  return {
    pass: false,
    message: () => (message ? message : 'fails by .toFail() assertion'),
  };
}

expect.extend({ toFail });

Example

// FUNCTION UNDER TEST
function invokeWithTry(callback) {
  try {
    return callback();
  } catch (err) {
    return err;
  }
}

// SETUP THE CUSTOM MATCHER
function toFail(_, message) {
  this.dontThrow();  // Just record the error when a test fails, no error throwing
  return {
    pass: false,
    message: () => (message ? message : 'fails by .toFail() assertion'),
  };
}

expect.extend({ toFail });

// TESTS
test("fail works in a try/catch", () => {
  expect().not.toFail("unexpected failure"); // Correctly passes
  const result = invokeWithTry(() => expect().toFail("expected failure")); // Correctly fails
  expect(result).toBeUndefined(); // Correctly passes
});

[danielo515]

Any chance this will be fixed in any new release?

[nealeu]

Hell... I've just arrived here to find a regression from >2.5 years ago and it's full of workarounds.

Will someone either close this as won't fix or fix it!

[joeskeen]

Or just move to another testing framework. I moved one project to vitest (Jest api compatible) this week, and noticed there was an expect.fail function.

[vtgn]

Hell... I've just arrived here to find a regression from >2.5 years ago and it's full of workarounds.

Will someone either close this as won't fix or fix it!

What a shame! This would be the worst answer to do not fix it! And yes you're right, it's pathetic that this regression on such a basic feature is more than 2,5 years old!! And being required to use workarounds is as pathetic too!

[vtgn]

Or just move to another testing framework. I moved one project to vitest (Jest api compatible) this week, and noticed there was an expect.fail function.

That's what I will do now: by letting this shit for a so long time, they completely discredited themselves and showed their total lack of seriousness. I will replace jest by another framework for my current and next projects for all the companies, and hardly advise to everybody to never choose that framework for their tests. I already do the same with TypeOrm which proved its total lack of seriousness too about the non fixed regressions and bugs.

[goncalvesnelson]

Don't take me wrong, but they don't owe you or us anything. The developers have their own lives, with their own problems and need to pay their bills at the end of the month too. If you really need this, you can use your spare time (like they do when they develop jest) and submit a PR to fix this issue.

[vtgn]

Don't take me wrong, but they don't owe you or us anything. The developers have their own lives, with their own problems and need to pay their bills at the end of the month too. If you really need this, you can use your spare time (like they do when they develop jest) and submit a PR to fix this issue.

That's what I do, but not on abandoned framework like here. I don't have time to waste either. Have a nice day!