[Bug]: Memory Leak with --runInBand

[dpchamps]

Version

27.4.7, ^28.0.3

Steps to reproduce

https://github.com/dpchamps/jest-oom-repro

To summarize the results of the repro:

--runInBand seems to prevent GC between specifications. While the native contexts repro might seem to be specifically related to jest-environment-jsdom, the v8 regex literal bug warrants an investigation into the test runner, as there could be some shared context that isn't properly getting cleaned up.

It's worth mentioning that the reason to suspect there is something shared at the jest test runner level (outside of say, jest-environment-jsdom) has to do with what surfaces the regex literal oom error. You can see a more targeted reproduction of this issue here: https://github.com/dpchamps/node-regex-oom. The issue seems to only be triggered by regex literals that have been used more than once. I wouldn't expect to see specifications that create new regex literals failing with this error, but when executed with --runInBand there appears to be a significant performance hit as more specifications are run, which will lead to the eventual Regex OOM error.

This happens with both jsdom and node environments.

Expected behavior

I expect to see isolated environments between specification runs to be properly garbage collected when run in band

Actual behavior

As outlined in the repro charts here, there is a memory leak.

Additional context

It might seem silly to demonstrate this repro against 1000 specifications, but we actually run into this problem at Walmart with a few suites that are quite large (not 1000 specs large, but still, we're feeling this pain).

These charts are included in the MVR, but will dupe them here for quick access

Environment

System:
    OS: macOS 12.0
    CPU: (10) arm64 Apple M1 Pro
  Binaries:
    Node: 16.15.0 - ~/.nvm/versions/node/v16.15.0/bin/node
    Yarn: 1.22.17 - ~/.nvm/versions/node/v16.6.2/bin/yarn
    npm: 8.5.5 - ~/.nvm/versions/node/v16.15.0/bin/npm
  npmPackages:
    jest: ^28.0.3 => 28.0.3

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[dpchamps]

This issue still exists, not stale

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[cdaringe]

go away stalebot

[cdaringe]

proactive stalebot hedge

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[dpchamps]

Not stale

[tchon]

It would be great to get acknowledgement from jest dev team so we can submit a patch.

[alande-amorim]

bump

[mikicho]

bump

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[cdaringe]

Bugger off, you stalebot bugger off! Ohhh bugger off, you stalebot buggger off!

https://youtu.be/o8Luriw7e2s

[jmoore240]

Not stale!!

[RuanAzevedo]

Issue reproducible with [email protected]

[jmoore240]

For those experiencing this issue, the following node and jest flags help keep the heap down by running the gc after tests. Memory leak is still there, but this is a work around that has been good enough for us in the meantime.

NodeJS Flags: --expose-gc --no-compilation-cache --max-old-space-size=700 Jest Flags: --no-cache --logHeapUsage

Hope this helps!

[fahminlb33]

Thanks @jmoore240, I have imlemented your workaround and it works like a charm. I also added some more options to my jest.config.js

{
  cache: false,
  maxWorkers: "50%",
  maxConcurrency: 3,
  transform: {
    "^.+\\.(t|j)s$": "@swc/jest"
  },
}

command: node --no-compilation-cache --expose-gc ./node_modules/jest/bin/jest.js

With this setup I can reduce the memory usage down from 8 GB to just 300 MB per test suite. Btw I used @swc/core to further speed up the test runner.

I also have stumbled upon other solution such as https://github.com/goloveychuk/fastest-jest-runner which doesn't involve calling jest from node with custom arguments, but I don't like adding new dependencies just for this issue.

  System:
    OS: Linux 5.15 Ubuntu 20.04.4 LTS (Focal Fossa)
    CPU: (12) x64 Intel(R) Core(TM) i5-10400 CPU @ 2.90GHz
  Binaries:
    Node: 18.12.1 - ~/.nvm/versions/node/v18.12.1/bin/node
    Yarn: 1.22.19 - ~/.local/share/pnpm/yarn
    npm: 9.2.0 - ~/.nvm/versions/node/v18.12.1/bin/npm
  npmPackages:
    jest: ^29.3.1 => 29.3.1

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 30 days with no activity. Please open a new issue if the issue is still relevant, linking to this one.

[HashemKhalifa]

I'm still facing the same issue btw!

[github-actions[bot]]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. Please note this issue tracker is not a help forum. We recommend using StackOverflow or our discord channel for questions.