[Bug]ReferenceError: __test is not defined

[VictorGlindasPaf]

I seem to get this error seemingly at random when running the tests. Following the stack trace just takes me to the very bottom of my story file.

 FAIL   browser: chromium  src/Tag.stories.tsx (26.779 s)
  ● Base Components/Tag › Variants › smoke-test

    page.evaluate: ReferenceError: __test is not defined

      at eval (eval at evaluate (:3:2389), <anonymous>:4:17)
      at t.default.evaluate (<anonymous>:3:2412)
      at t.default.<anonymous> (<anonymous>:1:44)
      at testFn (src/Tag.stories.tsx:84:37)
      at Object.<anonymous> (src/Tag.stories.tsx:99:17)

[yannbf]

Hey @VictorGlindasPaf thanks for opening this issue! Is that the only failure in your tests, or do previous tests fail too? Is there more info you can give to help us reproduce the issue?

[VictorGlindasPaf]

These errors were definitely mixed with other ones, but I'm not sure in which order they came.

I'll see if I have time to get some more info about this when back at work on Monday.

[valentinpalkovic]

@yannbf The error is flaky and happens occasionally. Yesterday, even in this repository, it happened once when I was running the tests. Reexecuting the tests made the error disappear. It seems to be a kind of timing issue. This happens occasionally on my private Gitlab pipeline as well (flaky). Locally it just happens rarely.

[yannbf]

@yannbf The error is flaky and happens occasionally. Yesterday, even in this repository, it happened once when I was running the tests. Reexecuting the tests made the error disappear. It seems to be a kind of timing issue. This happens occasionally on my private Gitlab pipeline as well (flaky). Locally it just happens rarely.

Interesting! Can we chat more about it? Can you reach out on discord at Yann#9096 or twitter?

[valentinpalkovic]

@VictorGlindasPaf I had the same issue. In my case, the issue was a redirection happening in the components by using a decorator, redirecting to an SSO page when a Token was not available in the localstorage. Maybe you have a similar issue and a redirection happens in special cases or you are triggering a page reload. This would explain, why the initial script, which contains the __test function, is not available in the head section anymore.

[janivo]

I'm having the same problem. The error seems to occur at random. I'm using the test-runner to take snapshots of stories of web components. Because the web components are not hydrated at the time the postRender hook runs, I added a waitForFunction to wait for my components to have the .hydrated class.

This is my test-runner.js file:

module.exports = {
  async postRender(page, context) {

    await page.waitForFunction(
      () => {
        const inoElements = Array.from(
          document.getElementsByTagName('*')
        ).filter((el) => el.tagName.startsWith('INO'));

        return inoElements.every((el) => el.classList.contains('hydrated'));
      },
      { polling: 50 }
    );

    await page.$('#root');

    const elementHandler = await page.$('#root');
    const innerHTML = await elementHandler.innerHTML();
    expect(innerHTML).toMatchSnapshot();
  },
};

The introduction of this snippet led to this error. Any ideas why this is happening?

[ericclemmons]

I can reproduce this by setting resetContextPerTest: true for https://github.com/playwright-community/jest-playwright.

[joriswitteman]

For me, this is caused when I navigate to a different URL trying to capture snapshots during postRender(). Probably the test runner injects something into the page that is lost by navigating to another URL.

(The use case for navigating for me, is to put different arg values in the URL and as such capture different permutations of the Story).

[gmwill934]

getting same error, following the docs, it says zero config, but do we need extra config?

[AnnaHoege]

I'm also getting this error while testing web components. It started after migrating to Storybook 7 and switching builders from webpack to vite.

Now tests will also fail at random, probably because vite takes too long for the test-runner to render components.

[yannbf]

Hey everyone! If you can share a minimal reproduction of this issue, I'd love to take a look at. I cannot get to reproduce this, I also tried setting resetContextPerTest: true as suggested by @ericclemmons.

@AnnaHoege given you have hit this issue recently, could you share a reproduction? Thank you!

[kasper573]

I wish I had a repro to share, but unfortunately I don't. I can only add another anecdote that this happens randomly when using storybook 7 and vite. I get this error in the CI every now and then when making arbitrary changes, and every time it fails I can make the same build pass by simply retrying in the CI without making any code changes. Feels like something is flaky inside of the storybook + vite integration.

[kasper573]

I have an optimistic update: I seem to have found a consistent (but unforunately not minimal) repro. I seem to get this error consistently for .stories files that are using the play function. In the code snippet below, if I keep the CanOnlySeeContentWhenOpen story, I get the error. If I remove it, I do not get the error.

This code snippet is from a large monorepo, so it could potentially be caused by something else too, but I figured I'd share what I have.

import { expect } from "@storybook/jest";
import type { Meta, StoryObj } from "@storybook/react";
import { userEvent, within } from "@storybook/testing-library";
import { useState } from "react";
import { Modal } from "./Modal";

export default {
  title: "atoms/Modal",
  component: Modal,
  tags: ["autodocs"],
} satisfies Meta<typeof Modal>;

export const Default: StoryObj<Meta<typeof Modal>> = {
  args: { children: "Hello World!", open: true },
};

export const CanOnlySeeContentWhenOpen: StoryObj<Meta<typeof Modal>> = {
  args: { children: "Hello World!" },
  render() {
    const [open, setOpen] = useState(false);
    return (
      <>
        <Modal open={open}>Hello World</Modal>
        <button onClick={() => setOpen(true)}>Show</button>
      </>
    );
  },
  async play({ canvasElement }) {
    const canvas = within(canvasElement);
    expect(canvas.getByText("Hello World")).not.toBeVisible();
    await userEvent.click(canvas.getByRole("button"));
    expect(canvas.getByText("Hello World")).toBeVisible();
  },
};

Here's my stack trace in case it helps:

FAIL browser: chromium src/atoms/Modal.stories.tsx (9.483 s)
    ● Console
  
      console.log
        An error occurred in the following story, most likely because of a navigation: "atoms/Modal/Default". Retrying...
  
        at Object.<anonymous> (src/atoms/Modal.stories.tsx)
  
    ● atoms/Modal › CanOnlySeeContentWhenOpen › smoke-test
  
      page.evaluate: ReferenceError: __test is not defined
  
        at eval (eval at evaluate (:202:30), <anonymous>:3:57)
        at UtilityScript.evaluate (<anonymous>:204:17)
        at UtilityScript.<anonymous> (<anonymous>:1:44)
        at src/atoms/Modal.stories.tsx:325:58
        at step (src/atoms/Modal.stories.tsx:109:23)
        at Object.next (src/atoms/Modal.stories.tsx:50:20)
        at asyncGeneratorStep (src/atoms/Modal.stories.tsx:4:28)
        at _next (src/atoms/Modal.stories.tsx:22:17)
        at src/atoms/Modal.stories.tsx:27:13
        at src/atoms/Modal.stories.tsx:19:16
        at testFn (src/atoms/Modal.stories.tsx:377:49)
        at Object.<anonymous> (src/atoms/Modal.stories.tsx:390:33)
        at step (src/atoms/Modal.stories.tsx:109:23)
        at Object.next (src/atoms/Modal.stories.tsx:50:20)
        at asyncGeneratorStep (src/atoms/Modal.stories.tsx:4:28)
        at _next (src/atoms/Modal.stories.tsx:22:17)
        at src/atoms/Modal.stories.tsx:27:13
        at Object.<anonymous> (src/atoms/Modal.stories.tsx:19:16)

My CI is github actions using ubuntu-latest

[yannbf]

Hey @ksandin!

Given the error message:

An error occurred in the following story, most likely because of a navigation: "atoms/Modal/Default". Retrying...

Are your stories doing any navigation on click? Navigations break the test-runner.

[kasper573]

@yannbf I have no navigations in the code that is being tested. As to why that warning appears, My guess is that something under the hood in storybook is causing a navigation.

[matttk]

Any update on this issue? Unfortunately, we just added storybook testing to our pipeline and ran into this issue multiple times today. I can't say much about a repro, as the issue only started when to started running the tests automatically - and we haven't written many tests yet. We are using storybook version 7.5.3 and test-runner 0.14.

[yannbf]

Hey everyone, it's really difficult to fix an issue when there is no reproduction available. I have been trying to reproduce this but without any success of. We use the test-runner daily in about 40 different projects, and so far have not seen this error. I recall this happening in very early versions of the test-runner, but not anymore. Please do share a reproduction if you have one, I'd be incredibly thankful for that!

[ttt43ttt]

I can reproduce this when using the workaround in https://github.com/mswjs/msw-storybook-addon/issues/82

// Yuck...
Loading.decorators = [
  (Story) => {
    useEffect(() => {
      return () => window.location.reload();
    }, []);

    return <Story />;
  },
];

[malixsys]

FYI: this happened to me when storybook was running in another terminal...

[kasper573]

Looking at the source code, this smells like a race condition to me.

The __test invocation is defined here:

https://github.com/storybookjs/test-runner/blob/6624c5e6b7958e05f7c9dbeb57b581e374202c2b/src/playwright/transformPlaywright.ts#L28-L30

And the __test definition is in setup-page-script.ts: https://github.com/storybookjs/test-runner/blob/6624c5e6b7958e05f7c9dbeb57b581e374202c2b/src/setup-page-script.ts#L266

Which in turn is loaded in setupPage: https://github.com/storybookjs/test-runner/blob/6624c5e6b7958e05f7c9dbeb57b581e374202c2b/src/setup-page.ts#L64-L75

Could the explanation be that there is a race between the invocation and loading of the setup page?

[abenhamdine]

This issue appears randomly for us also. We have nothing "special" in our stories, except we do a snapshot in postRender We don't click url, redirect or whatever

[throw5way]

It seems the race condition is resolved by running that initialization script not with evaluate, but with addInitScript.

diff --git a/node_modules/@storybook/test-runner/dist/playwright/transformPlaywright.js b/node_modules/@storybook/test-runner/dist/playwright/transformPlaywright.js
index f6b3bdf..328350c 100644
--- a/node_modules/@storybook/test-runner/dist/playwright/transformPlaywright.js
+++ b/node_modules/@storybook/test-runner/dist/playwright/transformPlaywright.js
@@ -57,7 +57,7 @@ const testPrefixer = (0, import_template.default)(`
           await globalThis.__sbPreRender(page, context);
         }

-        const result = await page.evaluate(({ id, hasPlayFn }) => __test(id, hasPlayFn), {
+        const result = await page.addInitScript(({ id, hasPlayFn }) => __test(id, hasPlayFn), {
           id: %%id%%,
         });

diff --git a/node_modules/@storybook/test-runner/dist/playwright/transformPlaywright.mjs b/node_modules/@storybook/test-runner/dist/playwright/transformPlaywright.mjs
index 08d0006..dfaec94 100644
--- a/node_modules/@storybook/test-runner/dist/playwright/transformPlaywright.mjs
+++ b/node_modules/@storybook/test-runner/dist/playwright/transformPlaywright.mjs
@@ -25,7 +25,7 @@ const testPrefixer = template(`
           await globalThis.__sbPreRender(page, context);
         }

-        const result = await page.evaluate(({ id, hasPlayFn }) => __test(id, hasPlayFn), {
+        const result = await page.addInitScript(({ id, hasPlayFn }) => __test(id, hasPlayFn), {
           id: %%id%%,
         });

[yannbf]

Hey there @throw5way thanks for the suggestion! I made a canary with the fix at #445, please try it out and give some feedback if you can. Thanks!

yarn add @storybook/[email protected]

[benhili]

@yannbf that fixes my issue! I have been scratching my head over this for a week.

[hans-lizihan]

for us, after upgrading the the canary version, all test pass even though they should be failing.

also we are trying todo some individual retries to deflake test runs.

We start to see this error after we add the setupFilesAfterEnv config in jest

// .storybook/test-runner-jest.config.js
import {getJestConfig} from '@storybook/test-runner';

// The default Jest configuration comes from @storybook/test-runner
const testRunnerConfig = getJestConfig();

// eslint-disable-next-line import/no-default-export
export default {
  ...testRunnerConfig,
  /** Add your own overrides below, and make sure
   *  to merge testRunnerConfig properties with your own
   * @see https://jestjs.io/docs/configuration
   */
  setupFilesAfterEnv: [`<rootDir>/.storybook/test-global-setup.ts`],
};

// global setup.ts
import {beforeEach, jest} from '@jest/globals';

beforeEach(() => {
  jest.retryTimes(3, {logErrorsBeforeRetry: true});
});

[yannbf]

Hey @hans-lizihan could you please share a reproduction? I'd like to take a deeper look if possible. For the others, could you please let me know if the canary solves your issues?

[blcoyote]

Hey @yannbf The canary build solves our issues as reproduced by https://github.com/storybookjs/test-runner/issues/68#issuecomment-1837872661 all the tests succeed after upgrading from 0.17.0. I'm awaiting the final release before upgrading our pileline though.

[yannbf]

Glad to hear that @blcoyote! It seems that it causes issues when the test-runner is used for visual regression testing though. I will have to take a deeper look into that. In the meantime, everyone else, please give the canary a try and give me feedback if possible. Thanks!

[unional]

I notice this still occurs on 0.18.1. Is the canary change included in 0.18?

And I'm running visual snapshot testing.

[yannbf]

Hey @unional it is not, it's only in that canary version so far. I need more feedback whether it introduces any issues. Is the canary working without issues for you?