[bug] jest.retryTimes does not work correctly.

[david-morris]

Describe the bug A clear and concise description of what the bug is.

If I import storybook/jest's jest and call jest.retryTimes(n) in an interaction test, the interaction test will fail with The story was missing when trying to access it. If I eject the config and add a setupFilesAfterEnv file with jest.retryTimes(n), all tests pass even if there are interaction tests which cannot succeed.

To Reproduce Steps to reproduce the behavior:

1. Eject config
2. add jest.retryTimes() to ejected config
3. add jest.retryTimes() to an interaction test

Expected behavior There should be a way (and ideally a way which matches the upstream API) to retry tests for situations like startup thrash on tests and test environments that mysteriously fail tests no matter how long you give them when it is the first test that the worker handles.

Screenshots

page.evaluate: StorybookTestRunnerError: 
    An error occurred in the following story. Access the link for full output:
    http://127.0.0.1:6006/?path=/story/geo-newanalysis-area--form&addonPanel=storybook/interactions/panel

    Message:
     The story was missing when trying to access it.


    --------------------------------------------------

    Browser logs:

    error: {}

      at Object.<anonymous> (<anonymous>:275:13)
      at http:/127.0.0.1:6006/sb-preview/runtime.js:4:92863
          at Array.forEach (<anonymous>)
      at Channel.handleEvent (http:/127.0.0.1:6006/sb-preview/runtime.js:4:92847)
      at handler (http:/127.0.0.1:6006/sb-preview/runtime.js:4:91935)
      at Channel.emit (http:/127.0.0.1:6006/sb-preview/runtime.js:4:91990)
      at PreviewWeb.renderStoryLoadingException (http:/127.0.0.1:6006/sb-preview/runtime.js:96:686)
      at PreviewWeb.renderSelection (http:/127.0.0.1:6006/sb-preview/runtime.js:94:3233)
      at libs/geo/features/src/components/AnalysisDetails/Area/AreaAnalysisDetails.stories.tsx:168:58
      at step (libs/geo/features/src/components/AnalysisDetails/Area/AreaAnalysisDetails.stories.tsx:109:23)
      at Object.next (libs/geo/features/src/components/AnalysisDetails/Area/AreaAnalysisDetails.stories.tsx:50:20)
      at asyncGeneratorStep (libs/geo/features/src/components/AnalysisDetails/Area/AreaAnalysisDetails.stories.tsx:4:28)
      at _next (libs/geo/features/src/components/AnalysisDetails/Area/AreaAnalysisDetails.stories.tsx:22:17)
      at libs/geo/features/src/components/AnalysisDetails/Area/AreaAnalysisDetails.stories.tsx:27:13
      at libs/geo/features/src/components/AnalysisDetails/Area/AreaAnalysisDetails.stories.tsx:19:16
      at testFn (libs/geo/features/src/components/AnalysisDetails/Area/AreaAnalysisDetails.stories.tsx:220:49)
      at Object.<anonymous> (libs/geo/features/src/components/AnalysisDetails/Area/AreaAnalysisDetails.stories.tsx:233:33)
      at step (libs/geo/features/src/components/AnalysisDetails/Area/AreaAnalysisDetails.stories.tsx:109:23)
      at Object.next (libs/geo/features/src/components/AnalysisDetails/Area/AreaAnalysisDetails.stories.tsx:50:20)
      at asyncGeneratorStep (libs/geo/features/src/components/AnalysisDetails/Area/AreaAnalysisDetails.stories.tsx:4:28)
      at _next (libs/geo/features/src/components/AnalysisDetails/Area/AreaAnalysisDetails.stories.tsx:22:17)
      at libs/geo/features/src/components/AnalysisDetails/Area/AreaAnalysisDetails.stories.tsx:27:13
      at Object.<anonymous> (libs/geo/features/src/components/AnalysisDetails/Area/AreaAnalysisDetails.stories.tsx:19:16)

Circus clearly recognizes that something has gone wrong 3 times: But then it passes the test anyway despite having seen that:

System

Environment Info:

  System:
    OS: Linux 5.15 Debian GNU/Linux 11 (bullseye) 11 (bullseye)
    CPU: (16) x64 AMD Ryzen 7 PRO 4750U with Radeon Graphics
  Binaries:
    Node: 18.16.1 - /usr/local/share/nvm/versions/node/v18.16.1/bin/node
    Yarn: 1.22.19 - /usr/local/bin/yarn
    npm: 9.5.1 - /usr/local/share/nvm/versions/node/v18.16.1/bin/npm
  npmPackages:
    @storybook/addon-essentials: 7.0.23 => 7.0.23 
    @storybook/addon-interactions: 7.0.23 => 7.0.23 
    @storybook/core-common: ^7.0.23 => 7.0.23 
    @storybook/core-server: 7.0.23 => 7.0.23 
    @storybook/jest: ^0.1.0 => 0.1.0 
    @storybook/nextjs: 7.0.23 => 7.0.23 
    @storybook/test-runner: ^0.11.0 => 0.11.0 
    @storybook/testing-library: ^0.2.0 => 0.2.0 

Additional context I'm not forgetting to keep around this addon's setupFilesAfterEnv:

  setupFilesAfterEnv: [
    "<rootDir>/libs/storybook/src/setup-jest.js",
    ...(defaultConfig.setupFilesAfterEnv ?? []),
  ],

I've tried adding mine both before and after the default ones. Also, my jest-circus is @29.5.0 and @28.1.3 according to yarn why. so that's should't have the upstream error.

[david-morris]

One workaround strategy that works better is to set different default timeouts (jest.setTimeout()) in the setup-jest file on CI. If you've getting intermittent failures in CI, it might be because the CI is slower.

[sw-tracker]

+1 on retry functionality

[zargold]

I think in the documentation for retryTimes they say it only works with the "default" Jest circus test runner?

So I guess the question is what would make this storybook test runner compatible with retry times. What does Jest circus have that this doesn't?

[david-morris]

With storybook 8.0 the testing infrastructure is changing over to vitest.

• Will this repo still be here? @storybook/jest and @storybook/testing-library are going away.
• Could that change solve this issue?

[sw-tracker]

Has anyone found a workaround for this? This has become quite a time issue for us on our pipelines. We have ended up retrying the whole suite of tests if the first run fails (usually just one test) which doubles up the test execution time.

[david-morris]

@sw-tracker we stabilize unstable tests, or comment them out if that's not possible.

[sw-tracker]

@david-morris maybe my question was misleading, it isnt so much unstable tests, but the first test that runs tends to fail. As you mentioned on your initial comment, it is most likely startup thrash.

[david-morris]

@sw-tracker Yeah, we didn't find a workaround with limited scope. We lightened up the tests, increased the global test timeout in our setup-jest.js, and reduced the number of parallel tests.

[sw-tracker]

I get a different error from you when I try to set the retry:

// test-runner-jest.config.js
const { getJestConfig } = require('@storybook/test-runner');

module.exports = {
  // The default configuration comes from @storybook/test-runner
  ...getJestConfig(),
  /** Add your own overrides below
   * @see https://jestjs.io/docs/configuration
   */
  testTimeout: 60_000,
  setupFilesAfterEnv: ['./setupTests.js'],
};

// setupTests.js
import { jest } from '@storybook/jest';

jest.retryTimes(2);

Error:

[david-morris]

@sw-tracker yeah, I don't think global config of retryTimes is possible currently.

So we have a test-runner-jest.config.js:

const { getJestConfig } = require("@storybook/test-runner")

const defaultConfig = getJestConfig()

/**
 * @type {import('@jest/types').Config.InitialOptions}
 */
module.exports = {
  // The default configuration comes from @storybook/test-runner
  ...defaultConfig,
  /** Add your own overrides below
   * @see https://jestjs.io/docs/configuration
   */
  setupFilesAfterEnv: [
    "<rootDir>/libs/storybook/src/setup-jest.js",
    ...(defaultConfig.setupFilesAfterEnv ?? []),
  ],
  bail: 5,
}

And since the bail option is ignored, we use a higher timeout in that setup-jest.js config:

// jest.retryTimes(2, { logErrorsBeforeRetry: true })
// jest retries are inexplicably broken despite a patch being written 2020?!
// https://github.com/storybookjs/test-runner/issues/337
// allow overrides to default timeout
// this is helpful for accounting for CI runners being slow
const timeout = process.env.JEST_DEFAULT_TIMEOUT ?? 20000
if (timeout) {
  jest.setTimeout(timeout)
}

and then we set the timeout to 40000 (40 seconds) in our CI workflow.