Possible memory leak after upgrade to Storybook 6.5.10

[jmmccota]

Hello, After upgrading to storybook 6.5.10 frequently I'm getting some kind of memory leak. I'm making in a react library with typescript.

I'm using few plugins

addons: [
'@storybook/addon-links',
'@storybook/addon-essentials',
'@storybook/addon-interactions',
'@storybook/preset-create-react-app',
'@storybook/addon-a11y'
],
framework: '@storybook/react',

Error:

<--- Last few GCs --->

[296651:0x649b000]    76524 ms: Scavenge 2043.8 (2082.5) -> 2042.7 (2087.0) MB, 5.2 / 0.0 ms  (average mu = 0.434, current mu = 0.367) allocation failure 
[296651:0x649b000]    77468 ms: Mark-sweep (reduce) 2046.1 (2087.0) -> 2044.6 (2082.5) MB, 536.5 / 0.0 ms  (+ 241.1 ms in 53 steps since start of marking, biggest step 7.1 ms, walltime since start of marking 816 ms) (average mu = 0.350, current mu = 0.251

<--- JS stacktrace --->
FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
 1: 0xb0a860 node::Abort() [/nodejs/16.16.0/bin/node]
 2: 0xa1c193 node::FatalError(char const*, char const*) [/nodejs/16.16.0/bin/node]
 3: 0xcf9a6e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/nodejs/16.16.0/bin/node]
 4: 0xcf9de7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/nodejs/16.16.0/bin/node]
 5: 0xeb1685  [/nodejs/16.16.0/bin/node]
 6: 0xec134d v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/nodejs/16.16.0/bin/node]
 7: 0xec404e v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/nodejs/16.16.0/bin/node]
 8: 0xe8558a v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/nodejs/16.16.0/bin/node]
 9: 0x11fe2d6 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/nodejs/16.16.0/bin/node]
10: 0x15f2d39  [/nodejs/16.16.0/bin/node]
RpcIpcMessagePortClosedError: Process 296651 exited [SIGABRT].
Issues checking service aborted - probably out of memory. Check the `memoryLimit` option in the ForkTsCheckerWebpackPlugin configuration.
If increasing the memory doesn't solve the issue, it's most probably a bug in the TypeScript or EsLint.
RpcIpcMessagePortClosedError: Process 296651 exited [SIGABRT].
Issues checking service aborted - probably out of memory. Check the `memoryLimit` option in the ForkTsCheckerWebpackPlugin configuration.
If increasing the memory doesn't solve the issue, it's most probably a bug in the TypeScript or EsLint.

My package.json libs

"dependencies": {
    "@emotion/react": "^11.8.2",
    "@emotion/styled": "^11.8.1",
    "@fontsource/open-sans": "^4.5.8",
    "@fortawesome/fontawesome-free": "^6.1.1",
    "@fortawesome/fontawesome-svg-core": "^6.1.1",
    "@fortawesome/free-brands-svg-icons": "^6.1.1",
    "@fortawesome/free-regular-svg-icons": "^6.1.1",
    "@fortawesome/free-solid-svg-icons": "^6.1.1",
    "@fortawesome/react-fontawesome": "^0.1.18",
    "@mui/material": "^5.5.1",
    "@mui/x-date-pickers": "^5.0.0-alpha.6",
    "@types/jsonwebtoken": "^8.5.8",
    "axios": "^0.27.2",
    "date-fns": "^2.28.0",
    "formik": "^2.2.9",
    "jsonwebtoken": "^8.5.1",
    "preact": "10.10.6", // I also tried with react version 17
    "react-error-boundary": "3.1.4",
    "web-vitals": "^2.1.4",
    "yup": "^0.32.11"
},
"peerDependencies": {
    "react": "npm:@preact/compat", // I also tried with react version 17
    "react-dom": "npm:@preact/compat", // I also tried with react version 17
    "react-router-dom": "^6.3.0"
},
"devDependencies": {
    "@storybook/addon-a11y": "^6.5.10",
    "@storybook/addon-actions": "^6.5.10",
    "@storybook/addon-essentials": "^6.5.10",
    "@storybook/addon-interactions": "^6.5.10",
    "@storybook/addon-links": "^6.5.10",
    "@storybook/builder-webpack5": "^6.5.10",
    "@storybook/manager-webpack5": "^6.5.10",
    "@storybook/node-logger": "^6.5.10",
    "@storybook/preset-create-react-app": "^4.1.2",
    "@storybook/react": "^6.5.10",
    "@storybook/testing-library": "^0.0.13",
    "@testing-library/jest-dom": "^5.16.2",
    "@testing-library/react": "^12.1.4",
    "@testing-library/user-event": "^14.1.1",
    "@types/jest": "^27.4.1",
    "@types/jest-axe": "^3.5.3",
    "@types/node": "^16.11.26",
    "@types/react": "^18.0.15 || ^17.0.40",
    "@types/react-dom": "^18.0.15 || ^17.0.13",
    "climem": "^1.0.3",
    "eslint": "^8.14.0",
    "eslint-plugin-jest-dom": "^4.0.1",
    "eslint-plugin-storybook": "^0.6.3",
    "eslint-plugin-testing-library": "^5.3.1",
    "husky": "^7.0.4",
    "jest-axe": "^6.0.0",
    "node-sass": "^7.0.1",
    "prettier": "^2.2.1",
    "react-refresh": "^0.14.0",
    "react-router-dom": "^6.3.0",
    "react-scripts": "5.0.0",
    "sass": "^1.49.9",
    "typescript": "^4.6.2",
    "webpack": "^5.70.0"
},

Some additional info, I also tried with other node and npm versions:

node -v v16.16.0

npm -v 8.11.0

[zhyd1997]

I guess it's node-sass problem, try to remove it because you already have sass module.

node-sass has deprecated, you have no reason to use it.

[LucaColonnello]

We're having the same issue when moving to pnpm in a lerna monorepo (previously yarn), and we also use node-sass, but I removed all scss imports and still have the same issue (also it worked using yarn, but then again maybe installing fresh with pnpm got some updated packages that are now causing leaks).

Here's the issue https://github.com/storybookjs/storybook/issues/18907

[LucaColonnello]

@jmmccota in my case, it only happens in dev mode, so I think it's the dev server. Can you try making a prod build of storybook and run it locally?

We have 90+ stories, but if we run with a couple of stories (none MDX), no memory leaks. As soon as we add all the stories (starting from just 50 actually), or even just 1 MDX story, the memory leaks starts to happen consistently (in dev mode only).

[jmmccota]

@jmmccota in my case, it only happens in dev mode, so I think it's the dev server. Can you try making a prod build of storybook and run it locally?

Hi @LucaColonnello, my prod build works fine, even my pipelines doesn't throw memory leaks. I'm not using MDX but I have around 70 stories and growing

[LucaColonnello]

Yep, I believe that's the dev server. Are you using yarn or pnpm @jmmccota ?

EDIT: ah it seems you're using npm, I haven't tried with npm 8, that's interesting, maybe it's something that affects both pnpm / npm but not yarn.

[jmmccota]

Are you using yarn or pnpm @jmmccota ?

@LucaColonnello I'm using NPM, I'll try with yarn to see what happens

[shilman]

Which version of Storybook are you upgrading from? The MDX implementation changed significantly in 6.5, so it's possible that a problem was introduced.

Do you a have a reproduction repo you can share? If not, can you create one? See how to create a repro. Thank you! 🙏

[jmmccota]

@shilman I've upgrade from 6.5.9. Even deleting package-lock, node_modules and npm cache, this issue keeps

[jmmccota]

@LucaColonnello I'm using NPM, I'll try with yarn to see what happens

@LucaColonnello same issue here

[LucaColonnello]

@shilman the main issue with these types of problems is that they occur on large monorepos, like in the issue I opened, which makes it very hard to create a reproduction codebase.

We need perhaps a bit more direction to be able to replicate this in isolation, which requires essentially debugging the problem. This is why I asked for help in my issue, so we can isolate, leading to a reproduction repo.

[zhyd1997]

Here is a good example that using pnpm and storybook: LedgerHQ/ledger-live.

when I upgraded the storybook version from 6.3.6 to 6.5.10, there was no memory leak issue.

# Terminal 1
pnpm install

# Terminal 2
cd libs/ui/packages/react

npx storybook upgrade
# ignore the error: `Unsupported URL Type "workspace:": workspace:^`

# Terminal 1
pnpm install

# Terminal 2
pnpm run storybook

Maybe the stories files were too small 👀 .

[LucaColonnello]

Thank you @zhyd1997 ! I tried upgrading now and it didn't help unfortunately...

Still have the memory leak! :'(

[zhyd1997]

@LucaColonnello

What I mean is Ledger Live has no memory leak issue, not sure what's happened.

[LucaColonnello]

Ah I see @zhyd1997 thank you! yeaaah we have something like 90+ stories, which is what's hard to reproduce as it works just fine if we halve the amount of stories we load.

[jmmccota]

With less stories works good, weird :(

[zhyd1997]

With less stories works good, weird :(

try binary search 👀 ?

[LucaColonnello]

I was able to narrow it down to watchpack watching directories. As soon as I disable the watcher (manually commenting it out), everything works just as fine.

More specifically, it's the directory watcher, not the file ones...

[jmmccota]

Hi @LucaColonnello is it storybook watcher? how did you do that?

[LucaColonnello]

Hey @jmmccota ! So basically I opened the watchpack package in my node_modules and commented the file and directory watchers instantiations. The code depends on what version of webpack you're using. As you're on v5, this might be a bit different for you though, but that's what I did...

[jmmccota]

Following this https://github.com/TypeStrong/fork-ts-checker-webpack-plugin/blob/main/README.md#options, I've added this

"fork-ts-checker": {
	"typescript": {
		"memoryLimit": 4096
	}
}

And the error is gone, for while

[LucaColonnello]

Interesting @jmmccota ! Thank you for sharing!

is this something you added to storybook’s webpack config?

I’m not on a TS project, but I’d assume this will still run anyway?

[jmmccota]

Hello @LucaColonnello

is this something you added to storybook’s webpack config?

You can do by webpack config as described in link.

I’m not on a TS project, but I’d assume this will still runs anyway?

I'm not sure, it worth a trial

[LucaColonnello]

Small update here, I've upgraded to webpack 5 and the leak is gone, but the loading takes minutes at best, whether I put 1 story or 90.

I'm not sure how to fix it, I need some direction in debugging locally what the performance issue seems to be. Does anybody know if there is a way to debug timing spent by storybook on different phases of the dev server process?

[LucaColonnello]

On the pnpm issue, the memory leak happened mainly because storybook is loading all the stories from node_modules too using the globby package.

I'll open a new issue for this though...