cypress icon indicating copy to clipboard operation
cypress copied to clipboard

Cypress crashes after getting around 2000mb of memory

Open Tomerlev01 opened this issue 1 year ago • 23 comments

Test code to reproduce

It happened to me "often" on some specs on the "cypress run" mode And in the "cypress open" it happened a lot The memory seems to increase from test to test and not to return to the "base levels"

This is the error that I got

<--- Last few GCs --->

[1419:0x44003e4000] 459478 ms: Mark-Compact 1950.0 (2075.7) -> 1933.0 (2061.9) MB, 1479.45 / 0.05 ms (average mu = 0.072, current mu = 0.016) allocation failure; GC in old space requested [1419:0x44003e4000] 460269 ms: Mark-Compact 1945.5 (2066.1) -> 1936.5 (2065.8) MB, 706.73 / 0.02 ms (average mu = 0.085, current mu = 0.106) allocation failure; scavenge might not succeed

<--- JS stacktrace --->

[1419:0124/185431.057257:ERROR:v8_initializer.cc(753)] V8 javascript OOM (Ineffective mark-compacts near heap limit).

image

Cypress Mode

both modes

Cypress Version

13.6.4

Browser Version

121.0.6167.139

Node version

v20.10.0

Operating System

14.3

Memory Debug Logs

No response

Other

Happened a lot, can't run a pull regression like this. Our project contain around 225 tests.

Tomerlev01 avatar Feb 08 '24 15:02 Tomerlev01

@Tomerlev01 Did you try setting experimentalMemoryManagement to true?

jennifer-shehane avatar Feb 08 '24 17:02 jennifer-shehane

@Tomerlev01 Did you try setting experimentalMemoryManagement to true?

Yeah, I tried that also, doesn't work. (This is my other account)

tomerlev001 avatar Feb 08 '24 20:02 tomerlev001

I got the same problems... cant run a single test in a pipeline :(

tklostermannNSD avatar Feb 09 '24 07:02 tklostermannNSD

I also have the same problem, my test would run for 10 minutes and then crash. I have split it up into smaller 'it' blocks and there are 7 of them and it still crashes! This is crazy, I see so many people with the same issue and there is no fix. I have tried both experimentalMemoryManagement and numTestKeptInMemory and nothing works.

I've just joined a new job and created a full regression test suite and it was going so well and now this is happening.

We need a fix.

bensouth avatar Feb 09 '24 16:02 bensouth

If anyone could provide a reproducible example, that would be helpful.

jennifer-shehane avatar Feb 09 '24 16:02 jennifer-shehane

I'm having this same issue on Cypress 13.6.4, 13.6.3 and 13.5.0 looks like could be something related to vite or node 20, the snapshots to keep tests in memory are storing an unusual amount of data and overflowing, if I set numTestsKeptInMemory to 1 and check the Memory consuming on Chrome I can see we are storing almost 4000mb, but if I set numTestsKeptInMemory to 0 we get just 150mb

CaiqueCoelho avatar Feb 09 '24 21:02 CaiqueCoelho

If anyone could provide a reproducible example, that would be helpful.

You can simulate it by running a simple test over and over. You will see that in every run the memory grows. I tested it with a simple "cy,visit("www.twitch.com") command Open cypress > run the same test over and over > observe the memory grow Eventually, the cypress crushed.

tomerlev001 avatar Feb 11 '24 18:02 tomerlev001

Does someone have maybe information about how to solve it? Maybe I need to provide more details? Please let me know.

Tomerlev01 avatar Feb 19 '24 14:02 Tomerlev01

You can try to add "numTestsKeptInMemory": 0, in config.json. It works after I set.

dyr136006 avatar Feb 21 '24 07:02 dyr136006

I tried it, but still, the crushing is reproduced. It happened on the "run" mode as well. (the 'numTestsKeptInMemory' doesn't effect this mode)

Tomerlev01 avatar Feb 21 '24 08:02 Tomerlev01

If anyone could provide a reproducible example, that would be helpful.

isn't using Windows with Electron browser enough? I noticed with each spec rising memory and crushing around 4 GB, it might be that my test had a lot of within() clauses and iterating over some items causing to store excessive amount of info (?)
I tried to use a reasonable numTestsKeptInMemory considering my spec has 12 tests, but it did not really help I seemed to have much less of those since I switched to test in Edge (Chrome seemed to work similar), somehow it splits processes to multiple threads and is separate from Cypress itself Note I am running on v12.17 and old node

goska-malaga avatar Feb 21 '24 10:02 goska-malaga

I found a workaround, and it does work which has made me very happy.

All you need to do is run your tests in --headed and it seems to reduce the amount of memory used (I know crazy) but it works and my pipelines are back up and running.

bensouth avatar Feb 21 '24 11:02 bensouth

As I said, this is happening to me when I'm using the headless mode (doing it by running the tests in the 'run' mode).

tomerlev001 avatar Feb 21 '24 19:02 tomerlev001

I found a workaround, and it does work which has made me very happy.

All you need to do is run your tests in --headed and it seems to reduce the amount of memory used (I know crazy) but it works and my pipelines are back up and running.

Thanks a lot for the hint! You've saved my day! As soon as I switched to headed mode our pipeline came back to normal! Cypress team should invest their time to this issue as this is clearly a performance bug.

viktorgogulenko avatar Feb 24 '24 16:02 viktorgogulenko

Yah, that's unusual. Chrome headless browser has some differences. On Cypress's size, we change things so that it uses LESS memory when in headless for sure, so I wonder if it's some difference with how Chrome --headless runs vs --headed. And also whether it's related to their headless=new changes.

A reproducible example would still be helpful.

jennifer-shehane avatar Feb 26 '24 20:02 jennifer-shehane

Im actually having this same issue. Trying with --headed mode on CI seems to work. However Im not being able to reproduce it locally. What I can affirm with certain, however, is that I was not having this issue with v13.6.1. It started happening after I updated, and even by splitting down the tests alot, it still crashes.

mateustalles avatar Feb 26 '24 23:02 mateustalles

Yah, that's unusual. Chrome headless browser has some differences. On Cypress's size, we change things so that it uses LESS memory when in headless for sure, so I wonder if it's some difference with how Chrome --headless runs vs --headed. And also whether it's related to their headless=new changes.

A reproducible example would still be helpful.

This is what I wrote above: You can simulate it by running a simple test over and over. You will see that in every run the memory grows. I tested it with a simple "cy.visit("www.twitch.com") command Open cypress > run the same test over and over > observe the memory grow Eventually, the cypress crushed.

Tomerlev01 avatar Feb 28 '24 07:02 Tomerlev01

We were facing memory crashes a lot, for us running with CYPRESS_NO_COMMAND_LOG=1 set helped.

memodi avatar Apr 17 '24 14:04 memodi

We were facing memory crashes a lot, for us running with CYPRESS_NO_COMMAND_LOG=1 set helped.

Interesting. I will try that command. Thanks!

Shame that I will not see the command log for investigation. But for a stable run, it could be worth it :)

Tomerlev01 avatar Apr 17 '24 15:04 Tomerlev01

None of the above solutions worked for my team. We have just a few Cypress E2E tests running in a Github Actions CI pipeline. It would error out with that same message in the OP during the execution of the third or fourth test, without fail.

Turns out our problem had to do with Cypress Cloud. Test Replay was turned on by default when we created our Cypress Cloud project, and that was evidently causing major issues in CI.

Turning off Test Replay in our Cypress Cloud project solved the issue.

ndirig avatar Apr 19 '24 18:04 ndirig

@ndirig Could you provide a link to a URL of a Test Replay in the Cloud where you were facing this issue?

jennifer-shehane avatar Apr 25 '24 18:04 jennifer-shehane

@ndirig Could you provide a link to a URL of a Test Replay in the Cloud where you were facing this issue?

@jennifer-shehane I'm afraid not. All the CI runs in which I got the FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory error are showing up in Cypress Cloud as "incomplete". Test replay button is disabled with the message "Test Replay is not available. This test timed out."

Not sure if this will do you any good, but here's the URL to the run itself in Cypress Cloud. https://cloud.cypress.io/projects/sf3837/runs/12/test-results

ndirig avatar Apr 26 '24 13:04 ndirig

@ndirig Could you provide the full terminal output? Providing a run with DEBUG log would be ideal to track this down though.

jennifer-shehane avatar Apr 26 '24 15:04 jennifer-shehane

None of the above solutions worked for my team. We have just a few Cypress E2E tests running in a Github Actions CI pipeline. It would error out with that same message in the OP during the execution of the third or fourth test, without fail.

Turns out our problem had to do with Cypress Cloud. Test Replay was turned on by default when we created our Cypress Cloud project, and that was evidently causing major issues in CI.

Turning off Test Replay in our Cypress Cloud project solved the issue.

This also seemed to work on our end (we use jenkins).

image

bjornlauwerijs avatar Aug 01 '24 12:08 bjornlauwerijs

I am also facing that issue with Cypress v13.7.0, the issue reproduces on the Cypress cloud. The only solution I have now is to split my specs, the spec holds 38 tests, so I split it into 18 and 20 tests in different specs. I have 617 tests in 57 specs.

asafk-biocatch avatar Aug 04 '24 20:08 asafk-biocatch

@bjornlauwerijs Please try upgrading to 13.13.2. We fixed a memory leak that was occuring with Test Replay turned on. We'd be interested to work with you on getting debug logs and memory logs if this is still occurring with Test Replay enabled after upgrading.

jennifer-shehane avatar Aug 05 '24 15:08 jennifer-shehane

@jennifer-shehane i have the exact same issue as mentioned by @bjornlauwerijs The cypress version is the latest 13.13.2. While doing tests and looking to the memory being used with the Test Replay ON, for some reason in some simple action or when the test fails, it crashes and freezes, it does not fail the test and stop nor continue to the next steps and the memory starts rising until it reaches max and it gives out of memory. Running the exact same tests under the same conditions but with Test Replay OFF, the test fails or gives an error on the same spot as it should, and continues running the remaining steps until the end, and the memory keeps stable on 700 to 800Mb, definitely not even close of reaching max memory

rmonteiro88 avatar Aug 07 '24 15:08 rmonteiro88

@rmonteiro88 We are investigating some memory related issues with Test Replay on. If you have more information to share around the run logs or reproducible examples you can reach out to [email protected] to share those with us.

jennifer-shehane avatar Aug 07 '24 15:08 jennifer-shehane