redwood
redwood copied to clipboard
redwoodjs
[Bug]: `@redwoodjs/testing/config/jest/api` causes a memory leak on api side tests
What's not working?
At Nous our Redwood test suite has become incredibly resource hungry, to the point that we can't actually run the entire suite on our local machines. We can just about run it on CI by splitting the work between multiple GHA jobs.
After some investigation it's clear that we have a huge memory leak. And one of the contributors seems to be @redwoodjs/testing/config/jest/api, which leaks memory even with the most trivial test suite.
Apologies if this is something you're already aware of - I searched for similar issues but no luck 🙈
Otherwise our experience with Redwood remains very positive, thank you team for all your hard work 🙏
How do we reproduce the bug?
https://github.com/project-nous/redwood-jest-memory-leak-repro
What's your environment? (If it applies)
No response
Are you interested in working on this?
- [X] I'm interested in working on this
From my own debugging, with the standard repro I get
PASS api api/src/7.test.ts (158 MB heap size)
PASS api api/src/14.test.ts (204 MB heap size)
PASS api api/src/6.test.ts (286 MB heap size)
PASS api api/src/15.test.ts (364 MB heap size)
PASS api api/src/17.test.ts (444 MB heap size)
PASS api api/src/4.test.ts (523 MB heap size)
PASS api api/src/20.test.ts (354 MB heap size)
PASS api api/src/16.test.ts (431 MB heap size)
PASS api api/src/5.test.ts (510 MB heap size)
PASS api api/src/13.test.ts (591 MB heap size)
PASS api api/src/0.test.ts (670 MB heap size)
PASS api api/src/9.test.ts (508 MB heap size)
PASS api api/src/8.test.ts (589 MB heap size)
PASS api api/src/12.test.ts (668 MB heap size)
PASS api api/src/1.test.ts (747 MB heap size)
PASS api api/src/19.test.ts (630 MB heap size)
PASS api api/src/3.test.ts (703 MB heap size)
PASS api api/src/10.test.ts (781 MB heap size)
PASS api api/src/2.test.ts (862 MB heap size)
PASS api api/src/11.test.ts (940 MB heap size)
PASS api api/src/18.test.ts (709 MB heap size)
Test Suites: 21 passed, 21 total
Tests: 21 passed, 21 total
Snapshots: 0 total
Time: 12.48 s
If I comment everything out in node_modules/@redwoodjs/testing/config/jest/api/jest.setup.js, I get:
PASS api api/src/7.test.ts (75 MB heap size)
PASS api api/src/18.test.ts (84 MB heap size)
PASS api api/src/20.test.ts (83 MB heap size)
PASS api api/src/19.test.ts (91 MB heap size)
PASS api api/src/2.test.ts (92 MB heap size)
PASS api api/src/15.test.ts (85 MB heap size)
PASS api api/src/9.test.ts (92 MB heap size)
PASS api api/src/14.test.ts (92 MB heap size)
PASS api api/src/13.test.ts (100 MB heap size)
PASS api api/src/12.test.ts (90 MB heap size)
PASS api api/src/3.test.ts (98 MB heap size)
PASS api api/src/11.test.ts (99 MB heap size)
PASS api api/src/1.test.ts (106 MB heap size)
PASS api api/src/10.test.ts (108 MB heap size)
PASS api api/src/4.test.ts (115 MB heap size)
PASS api api/src/0.test.ts (98 MB heap size)
PASS api api/src/16.test.ts (106 MB heap size)
PASS api api/src/8.test.ts (105 MB heap size)
PASS api api/src/17.test.ts (113 MB heap size)
PASS api api/src/6.test.ts (115 MB heap size)
PASS api api/src/5.test.ts (123 MB heap size)
Test Suites: 21 passed, 21 total
Tests: 21 passed, 21 total
Snapshots: 0 total
Time: 1.089 s, estimated 13 s
If I just comment out the beforeAll / afterAll / afterEach hooks in node_modules/@redwoodjs/testing/config/jest/api/jest.setup.js, I get:
PASS api api/src/7.test.ts (131 MB heap size)
PASS api api/src/1.test.ts (196 MB heap size)
PASS api api/src/19.test.ts (261 MB heap size)
PASS api api/src/6.test.ts (321 MB heap size)
PASS api api/src/4.test.ts (380 MB heap size)
PASS api api/src/3.test.ts (446 MB heap size)
PASS api api/src/8.test.ts (297 MB heap size)
PASS api api/src/9.test.ts (356 MB heap size)
PASS api api/src/13.test.ts (421 MB heap size)
PASS api api/src/12.test.ts (481 MB heap size)
PASS api api/src/16.test.ts (547 MB heap size)
PASS api api/src/5.test.ts (410 MB heap size)
PASS api api/src/20.test.ts (471 MB heap size)
PASS api api/src/14.test.ts (535 MB heap size)
PASS api api/src/10.test.ts (594 MB heap size)
PASS api api/src/17.test.ts (659 MB heap size)
PASS api api/src/15.test.ts (520 MB heap size)
PASS api api/src/18.test.ts (583 MB heap size)
PASS api api/src/11.test.ts (643 MB heap size)
PASS api api/src/2.test.ts (707 MB heap size)
PASS api api/src/0.test.ts (767 MB heap size)
Test Suites: 21 passed, 21 total
Tests: 21 passed, 21 total
Snapshots: 0 total
Time: 9.762 s, estimated 11 s
If I keep the beforeAll / afterAll / afterEach hooks in node_modules/@redwoodjs/testing/config/jest/api/jest.setup.js commented out and update some of the imports, namely:
-const { getConfig, getDMMF } = require('@prisma/sdk')
+const { getConfig, getDMMF } = require('@prisma/sdk/dist/engine-commands')
-const { setContext } = require('@redwoodjs/graphql-server')
+const { setContext } = require('@redwoodjs/graphql-server/dist/globalContext')
const { getPaths } = require('@redwoodjs/internal/dist/paths')
-const { defineScenario } = require('@redwoodjs/testing/dist/api')
+
+const { defineScenario } = require('../../../dist/api/scenario')
I get:
PASS api api/src/7.test.ts (94 MB heap size)
PASS api api/src/8.test.ts (113 MB heap size)
PASS api api/src/19.test.ts (114 MB heap size)
PASS api api/src/16.test.ts (134 MB heap size)
PASS api api/src/5.test.ts (153 MB heap size)
PASS api api/src/12.test.ts (137 MB heap size)
PASS api api/src/2.test.ts (157 MB heap size)
PASS api api/src/4.test.ts (174 MB heap size)
PASS api api/src/10.test.ts (192 MB heap size)
PASS api api/src/14.test.ts (210 MB heap size)
PASS api api/src/6.test.ts (228 MB heap size)
PASS api api/src/9.test.ts (246 MB heap size)
PASS api api/src/1.test.ts (264 MB heap size)
PASS api api/src/15.test.ts (282 MB heap size)
PASS api api/src/11.test.ts (300 MB heap size)
PASS api api/src/17.test.ts (318 MB heap size)
PASS api api/src/20.test.ts (336 MB heap size)
PASS api api/src/13.test.ts (354 MB heap size)
PASS api api/src/18.test.ts (372 MB heap size)
PASS api api/src/0.test.ts (390 MB heap size)
PASS api api/src/3.test.ts (408 MB heap size)
Test Suites: 21 passed, 21 total
Tests: 21 passed, 21 total
Snapshots: 0 total
Time: 3.174 s, estimated 4 s
i.e. heap is not increasing at such a fast pace and the tests are running significantly faster due to smaller import sizes (i.e. not importing the whole of the Prisma SDK.
Not sure if it's related, but it's also slightly alarming that this trivial test suite is taking >10s 😅
If I keep the
beforeAll/afterAll/afterEachhooks innode_modules/@redwoodjs/testing/config/jest/api/jest.setup.jscommented out and update some of the imports, namely:-const { getConfig, getDMMF } = require('@prisma/sdk') +const { getConfig, getDMMF } = require('@prisma/sdk/dist/engine-commands') -const { setContext } = require('@redwoodjs/graphql-server') +const { setContext } = require('@redwoodjs/graphql-server/dist/globalContext') const { getPaths } = require('@redwoodjs/internal/dist/paths') -const { defineScenario } = require('@redwoodjs/testing/dist/api') + +const { defineScenario } = require('../../../dist/api/scenario')I get:
PASS api api/src/7.test.ts (94 MB heap size) PASS api api/src/8.test.ts (113 MB heap size) PASS api api/src/19.test.ts (114 MB heap size) PASS api api/src/16.test.ts (134 MB heap size) PASS api api/src/5.test.ts (153 MB heap size) PASS api api/src/12.test.ts (137 MB heap size) PASS api api/src/2.test.ts (157 MB heap size) PASS api api/src/4.test.ts (174 MB heap size) PASS api api/src/10.test.ts (192 MB heap size) PASS api api/src/14.test.ts (210 MB heap size) PASS api api/src/6.test.ts (228 MB heap size) PASS api api/src/9.test.ts (246 MB heap size) PASS api api/src/1.test.ts (264 MB heap size) PASS api api/src/15.test.ts (282 MB heap size) PASS api api/src/11.test.ts (300 MB heap size) PASS api api/src/17.test.ts (318 MB heap size) PASS api api/src/20.test.ts (336 MB heap size) PASS api api/src/13.test.ts (354 MB heap size) PASS api api/src/18.test.ts (372 MB heap size) PASS api api/src/0.test.ts (390 MB heap size) PASS api api/src/3.test.ts (408 MB heap size) Test Suites: 21 passed, 21 total Tests: 21 passed, 21 total Snapshots: 0 total Time: 3.174 s, estimated 4 si.e. heap is not increasing at such a fast pace and the tests are running significantly faster due to smaller import sizes (i.e. not importing the whole of the Prisma SDK.
FYI, if I put the beforeAll / afterAll / afterEach hooks in node_modules/@redwoodjs/testing/config/jest/api/jest.setup.js back in but keep the import updates, I get:
PASS api api/src/7.test.ts (114 MB heap size)
PASS api api/src/8.test.ts (161 MB heap size)
PASS api api/src/19.test.ts (196 MB heap size)
PASS api api/src/12.test.ts (240 MB heap size)
PASS api api/src/9.test.ts (287 MB heap size)
PASS api api/src/18.test.ts (333 MB heap size)
PASS api api/src/16.test.ts (380 MB heap size)
PASS api api/src/14.test.ts (426 MB heap size)
PASS api api/src/4.test.ts (472 MB heap size)
PASS api api/src/5.test.ts (307 MB heap size)
PASS api api/src/10.test.ts (353 MB heap size)
PASS api api/src/20.test.ts (399 MB heap size)
PASS api api/src/0.test.ts (445 MB heap size)
PASS api api/src/1.test.ts (492 MB heap size)
PASS api api/src/3.test.ts (539 MB heap size)
PASS api api/src/2.test.ts (585 MB heap size)
PASS api api/src/15.test.ts (631 MB heap size)
PASS api api/src/13.test.ts (472 MB heap size)
PASS api api/src/6.test.ts (519 MB heap size)
PASS api api/src/11.test.ts (565 MB heap size)
PASS api api/src/17.test.ts (611 MB heap size)
Test Suites: 21 passed, 21 total
Tests: 21 passed, 21 total
Snapshots: 0 total
Time: 5.917 s
i.e. heap still out of control, but tests are significantly quicker
Hello both - thank you for reporting this and the reproduction.
I just merged in a few changes (mainly targetting the web side) in https://github.com/redwoodjs/redwood/pull/6281 - would you like to try this out in your real project to see if it improves things for you?
yarn rw upgrade -t canary - we're very close to a v3 release and the canary isn't quite ready for prime time yet, but would be a great if you could give it a try.
I'll try it in your reproduction in the mean time.
Hey so spent some time trying to improve speed/memory here - but I want to point out something important - the reproduction doesn't actually have any scenarios, or real usecases.
It's a tradeoff between being able to write scenarios, and seed the db without more boilerplate in each test vs keeping jest as vanilla as possible.
Adding this one to the current cycle just so we keep seeing it and thinking about next steps.
hey @dac09 @jtoar is this being looked into please? memory leak is killing our apps test and our CI bills. We have to introduce excessive (and expensive) parallelism in order to have small enough test suites, so that they don't eat away all the memory. This is a major issue for us and causes so much pain that we don't really consider redwood test helpers production ready and are considering moving away from it unless this gets resolved
@jan-stehlik it's still on our radar, but progress is sporadic right now. Preparing for v5 (react 18) is taking most of our attention; at first it seemed like it would take a long time but things are looking optimistic, so our attention may return to this fairly soon.
We've heard many complaints about the speed but not as many about the memory. I know those are related to some extent. Unless you've already connected with one of us, maybe it's worth connecting to see what you're doing in your project and if we can offer any tips to alleviate things for the time being?