Flaky tests reported for React 18 concurrent mode

[sidrak19]

• @testing-library/react version: ^14.0.0
• Testing Framework and version: Jest: ^29.5.0
• DOM Environment: jest-environment-jsdom: ^29.5.0 node: 16.10.0

Problem description:

When we're running tests on React 18 concurrent mode, we get a lot of flaky tests. These tests can have different reasons for failure, including:

• Delayed updates to the UI, which causes assertions to fail.
• Sometimes elements are not available on the screen (some of these can be fixed with replacing screen.getBy* with await screen.findBy*)
• Form updates are often delayed and tests fail (we use formik for implementing our forms).

Using React 18 without concurrent mode works fine and we don't see test flakiness without concurrent mode.

Have others encountered frequent test flakiness with React 18 concurrent mode? How did you resolve them?

[sidrak19]

Hi @mpeyper, could you please take a look? thanks!

[mpeyper]

Sorry @sidrak19, I maintain a related project (react-hooks-testing-library) and don't really have anything to do with maintaining this project.

[MatanBobi]

Hi @sidrak19, thanks for opening this one. It doesn't look like this bug report has enough info for one of us to reproduce it. Please provide a CodeSandbox (https://react.new/), or a link to a repository on GitHub. Here are some tips for providing a minimal example: https://stackoverflow.com/help/mcve

[sidrak19]

@MatanBobi Thanks for your response! I'm finding it difficult to reproduce this since our tests generally pass locally, and they are flaky on the CI builds. Can something on a high-load CI build environment cause flakiness?

Types of flaky issues I've seen include:

• userEvent.type misses characters in the string
• expect(element).toBeInTheDocument() ... element could not be found in the document
• Buttons which should be enabled are not.
• Buttons which should be disabled are not.

Many times I'm able to fix these by wrapping the assertions inside await waitFor, etc. I'm fine with making these fixes, but is there a way to reliably know all the errors? since these tests are flaky, one or the other randomly fails, and we're unable to switch to concurrent mode because of it.

[timkindberg]

For flaky tests you sometimes have to put the box under CPU load to reproduce locally, try this out. It runs one test file 30 times in parallel.

TESTFILE="{{file_path}}"
for INDEX in {1..30}
do
  npm run test -- $TESTFILE &
done
wait

[devhelpr]

I am also having these issues sometimes.. and can't reproduce it everytime (it's on private in-company repo's, so I can't share). This seems to help a little bit:

import { configure } from "@testing-library/react";
configure({ asyncUtilTimeout: 5000 });

I've added this to the jest setup via setupFilesAfterEnv in jest.config. This isn't a true fix offcourse...

[sidrak19]

thanks @devhelpr , I've also set asyncUtilTimeout to 10000, and it does help, but doesn't fix everything.

[devhelpr]

When I run the tests using runInBand , the chance seems to increase that they run succesfully.

In my case , the project is setup as a monorepo using lerna and yarn workspaces. There are multiple packages which have forms which are build using the react-hook-form library and jest and Testing-library are used for testing. When a test is run in isolation, directly started from vs.code, they run fine.

• testing-library/react 14.0.0
• testing-library/jest-dom 6.1.3
• jest 29.7..0
• jest-environment-jsdom 29.7.0
• react 18.2
• node 16.19

Since I can't share the code base, and it seems that a bigger code-base with lots of tests is needed to reproduce the issue, I would be happy to jump in a teams or zoom call with a maintainer to show the problem and help debug it.

[PupoSDC]

Our team is currently struggling with this issue as well. Tests are particularly flaky when running in the CI pipeline. We can, to some extent, reproduce the flakiness of the tests by running multiple runs of tests at the same time, saturating our machines, similar to what @timkindberg suggested.

Also, exactly like @sidrak19 reported, the bulk of our issues relate to either type events, or finding documents in the DOM.

We got into the habit of wrapping all our expect().toBeIntheDocument queries with waitFor... but in theory these should not be needed, and they do not always work, for example:

      await userEvent.click(within(rightDrawer).getByText('save'));
      // the drawer gets removed as a direct consequence of clicking the drawer
      // The code is very flaky without the waitFor, and still fails, albeit rarely with
      // the waitFor
      await waitFor(() => {
        expect(rightDrawer).not.toBeInTheDocument();
      });

In general this happens when we have BIG components being tested (a couple of hundred DOM elements, upwards of a thousand React elements). We are also using msw and there is a bigger predisposition of tests with MSW to fail more often. We are also using vitest instead of jest.

Unfortunately, since our code is proprietary, it's a bit tricky for us to share a whole test case, but I will try to put some effort into trying to reproduce it. But similarly to @devhelpr a show and tell over zoom may be possible to arrange.

[devhelpr]

If somebody has got an idea on how to fix this in testing-library itself, I am more then willing to test the change locally .. but also if it helps to gather some extra debug information... then please let me know, kind regards, Maikel

[sidrak19]

Hi @MatanBobi , do you have any suggestions for fixing these issues?

[ahilke]

TL;DR: Try to completely remove and re-install @testing-library dependencies.

I made an interesting discovery, sharing it here in case it helps somebody else, though I'm not sure it's the same issue others are experiencing. We have this issue of inexplicably flaky tests for a while now, but recently it became much worse. We finally figured out that we have two classes of flaky tests:

1. Tests that throw act warnings, an example warning can be seen below - interestingly these warnings originate from Material UI and it's styling engine (styled-components in our case, but I was able to reproduce it with emotion as well). These tests were the cause of increase in flaky tests we saw recently, and we were able to fix them by completely re-installing our @testing-library dependencies, i.e. yarn remove @testing-library/dom && yarn add @testing-library/dom. I don't really know why, but I have two theories:

   • this pulled in some updates of indirect dependencies that fixed the issue
   • this made sure all our projects in our mono-repo setup use the same version (our monorepo setup might not be perfectly isolated)

2. Tests that do not create act warnings. Sadly those still persist for us. They are what @PupoSDC described above.

Sadly we were not able to reproduce these outside our project yet, so that's all I can share for now. But maybe it helps someone, because I only stumbled upon this "fix" by chance.

Finally, here an example act warning for the first type of test failures:

Warning: An update to ForwardRef(FormControl) inside a test was not wrapped in act(...).
When testing, code that causes React state updates should be wrapped into act(...):
act(() => {
  /* fire events that update state */
});
/* assert on the output */
This ensures that you're testing the behavior the user would see in the browser. Learn more at https://reactjs.org/link/wrap-tests-with-act
    at FormControl (****/node_modules/@mui/material/node/FormControl/FormControl.js:92:44)
    at I (****/node_modules/styled-components/dist/styled-components.cjs.js:1:19114)
    at TextField (****/node_modules/@mui/material/node/TextField/TextField.js:83:44)
    at div
    at I (****/node_modules/styled-components/dist/styled-components.cjs.js:1:19114)
    at Grid (****/node_modules/@mui/system/Stack/createStack.js:147:24)
    at I (****/node_modules/styled-components/dist/styled-components.cjs.js:1:19114)
    at MultiInputDateRangeField (****/node_modules/@mui/x-date-pickers-pro/node/MultiInputDateRangeField/MultiInputDateRangeField.js:59:48)
    at LocalizationProvider (****/node_modules/@mui/x-date-pickers/node/LocalizationProvider/LocalizationProvider.js:24:19)
    at MobileDateRangePicker (****/node_modules/@mui/x-date-pickers-pro/node/MobileDateRangePicker/MobileDateRangePicker.js:24:75)
    at ****
    at div
    at I (****/node_modules/styled-components/dist/styled-components.cjs.js:1:19114)
    at div
    at I (****/node_modules/styled-components/dist/styled-components.cjs.js:1:19114)
    at div
    at I (****/node_modules/styled-components/dist/styled-components.cjs.js:1:19114)
    at div
    at I (****/node_modules/styled-components/dist/styled-components.cjs.js:1:19114)
    at Transition (****/node_modules/react-transition-group/cjs/Transition.js:135:30)
    at Collapse (****/node_modules/@mui/material/node/Collapse/Collapse.js:103:44)
    at div
    at I (****/node_modules/styled-components/dist/styled-components.cjs.js:1:19114)
    at ****
    at ****
    at ****
    at ****
    at ****
    at ****
    at ****
    at ****
    at ****
    at ****
    at ****

[sidrak19]

Hi everyone, there is something that really helped reduce the flakiness of our tests. We ensured that after every call of jest.useFakeTimers, we called jest.runOnlyPendingTimers(); and jest.useRealTimers();. This reduced flakiness a lot.

[DonikaV]

I updated "@testing-library/react": "^14.1.2",

And now all my tests are failing due to

 console.error
    Warning: An update to Component inside a test was not wrapped in act(...).
    
    When testing, code that causes React state updates should be wrapped into act(...):
    
    act(() => {
      /* fire events that update state */
    });
    /* assert on the output */

I am wrapping to act and nothing helps, re-installed all packages and still same error

[bredelman]

I'm also having this issue. In general increasing the waitFor timeout has been helpful but still have strange issues around userEvent.type where seemingly random characters appear in the input and so the tests end up failing. I am able to reproduce in my setup locally (this was only happening for us in CI randomly).

seq 8 | parallel -n0 CI=true npm test will run the tests concurrently and simulate the CPU overload that seems to be the issue in CI occasionally.

We're using vitest and so I also set the retry argument for vitest so that it will retry failed tests when they crop up. This seems to be helpful as well.

[liorpevzner]

I'm also having this issue. In general increasing the waitFor timeout has been helpful but still have strange issues around userEvent.type where seemingly random characters appear in the input and so the tests end up failing. I am able to reproduce in my setup locally (this was only happening for us in CI randomly).

seq 8 | parallel -n0 CI=true npm test will run the tests concurrently and simulate the CPU overload that seems to be the issue in CI occasionally.

We're using vitest and so I also set the retry argument for vitest so that it will retry failed tests when they crop up. This seems to be helpful as well.

Thanks for your comment. What do you mean retry argument? im also using vitest. Did you add something to the vitest config file to retry failed tests X amount of times?

[bredelman]

Thanks for your comment. What do you mean retry argument? im also using vitest. Did you add something to the vitest config file to retry failed tests X amount of times?

In vitest you can add an argument --retry=1 This PR added it in June

This argument has been super helpful but the issue still occurs (just more rare than before).

[jebbard]

I face the same issue when I upgrade from 13.4.0 to 14.x. Basically. I see the following effects:

• I see failures in the same test when I run all my test suites at ones.
• If I run individual tests or test suites, they are all passing.
• I have the following failure cases:
• expect(x).toBeInTheDocument() fails
• Timeout of a test is exceeded. Even if I double or quadruple the timeout, it still fails.
• Sometimes the whole test execution just hangs, never to complete.

[eps1lon]

I'm closing this since it just accumulates all sorts of possibly unrelated issues. Without a full reproduction it's impossible to tell what the issue is. It's very likely that the test or implementation was already problematic before and React 18 just flushed it out.

I'd love to help but need a reproduction (e.g. a repo to clone and command to run). The smaller the repro, the faster we can help. Please file a new issue with a concrete reproduction.