Cypress Open Crashing Chrome with "Aw Snap" after upgrade to Chrome 100

[dwaynegreen]

Current behavior

When running Cypress Open with multiple tests, Chrome 100 is more likely to crash with an "Aw, Snap" error message than previous versions of Chrome:

I was able to determine that Chrome 100 is more likely to crash by using a Chromium version from https://chromium.cypress.io/. I was then able to run the standard Cypress Tests examples with Chrome 99.0.4844.0 by specifying "npx cypress open --browser "pathToChromiumVersion\chrome.exe"

Here is three screenshots from running cypress open showing memory footprint with Chrome 99 using the Cypress Examples:

Here is three screenshots from running cypress open showing memory footprint with Chrome 100 using the Cypress Examples:

Desired behavior

I believe that Cypress should be able to keep a similar memory footprint between Chrome 99 and Chrome 100 or provide a better way to garbage collect between tests. I have looked at using numTestsKeptInMemory (https://docs.cypress.io/guides/references/configuration#Options) but we only have 14 tests in our build verification suite.

I would also like to have the numTestsKeptInMemory configurable to only keep tests that have failed in memory instead of the last X tests.

Test code to reproduce

Using Cypress Version 9.5.4 (as well as other versions), Cypress Examples that are installed with cypress on npm install -D cypress

Cypress Version

9.5.4

Other

No response

[rockhold]

@davidmunechika, did you have a chance to reproduce this against the 10.0-release branch? Just curious if it'll still be an issue before long.

[BlueWinds]

@rockhold - I've confirmed this for both 9.x and 10.x. Chrome/Chromium 100/101 with Cypress seems to use more memory than previous versions.

I did a bit of searching around Chromium bugs, and the best I've found is https://bugs.chromium.org/p/chromium/issues/detail?id=1309201#c9, which seems to indicate that the Chrome team is aware of some ongoing memory issues introduced around Chrome 100 and not yet resolved in 101; I don't know enough of the internals to make complete sense of this conversation.

With that said, I suspect that this in a Chromium regression, which Cypress will be powerless to resolve. :( Perhaps 102 or 103 will contain more solutions.

[BlueWinds]

The issue they mention related to extensions::Event::DeepCopy in particular seems like a likely culprit - Cypress is deeply bound up with browser events, and we create a ton of event for our internal use, far more than most "real sites" ever would. If there's a memory leak there, Cypress is perfectly positioned to pound it repeatedly.

[brian-mann]

If that's the case then maybe we shouldn't update the embedded Chromium in Electron to v100 yet...

@BlueWinds can you check our perf charts and see if there's anything egregious in there

[mjhenkes]

@brian-mann, we have already updated chromium in electron to v100 https://github.com/cypress-io/cypress/pull/21418

[extracker-ppetrov]

I can still reproduce this on Cypress 10.0.3 using Chrome. My "tests kept in memory" setting is set to 2. It crashes if I run more than 2 tests in a single run.

If the test kept in memory is set to 0, it is all good, no crashes with any browser. If the "tests kept in memory" setting is set to 2 and the browser is set to Electron, it is also all good, no crashes.

[tbiethman]

@extracker-ppetrov as of Cypress 10.4.0, the Electron browser bundled with Cypress is using Chromium 102. I'm curious if you're still experiencing these crashes with that version (or with the latest stable Chrome version 104)?

[JessefSpecialisterren]

@tbiethman I'm not extracker-ppetrov, but our team has also been facing lots of Aw Snap crashes lately, and I've been investigating the issue. Some results running a specific memory-hungry test file with Cypress 10.4.0 on Windows 10:

Chrome 104: consistent Aw Snap during the third test. Memory usage of Chrome is around 3gb right before the crash, my system has >5gb of available RAM at that point. It is unclear why Chrome doesn't just use more RAM instead of crashing Electron 102: completes the test file without issue. The interesting thing: memory usage is very similar to Chrome 104, but Electron seems to have no trouble using more than 3gb of RAM. I've seen Electron run without crashing with over 5gb of memory usage, while the largest footprint I've seen with Chrome 104 was 3.3gb (and then it crashed) Chromium 99: completes the test file without issue. The memory footprint has some interesting properties: the usage right after garbage collection doesn't differ that much between the 3 browsers, but Chromium 99's peak memory usage is significantly lower. Chromium 99 stayed below 2.6gb of memory usage during the entire test file, whereas Electron 102 peaked above 4gb and Chrome 104 crashed at around 3gb

Note: I have also experimented with --expose-gc and found that periodic forced garbage collection would delay the crash with Chrome 104, but I still couldn't complete the entire test file. Once memory usage starts approaching 3gb, Chrome 104 gets very likely to crash on my system, garbage collection or no

Note 2: I wondered if Chromium 99 would crash when memory usage got around 3gb, so I ran the same test file twice without closing the browser between runs and saw a peak memory usage of just over 4gb without issues

[succinct]

@tbiethman I'm also experiencing crashes with Cypress 10.4.0 and Chrome 104.0.5112.79, although my error code is SIGTRAP rather than a memory error.

[hodbby]

I use Cypress 10.6.0 Chrome v104 Electron v100 Both browsers crash as mentioned above. Only 3 weeks ago they were fine.

I cleaned the memory cache, tests kept in memory=1 None helped

Any idea?

[extracker-ppetrov]

@tbiethman I am still experiencing this. I am currently with Cypress 10.6.0 and Chrome 104. At this point even if my "tests kept in memory" is set to 0, I am seeing the issue, when my tests fail and rerun multiple times. I would suggest you try with a test suite that has multiple test cases in it and make them all fail. Stop and restart them multiple times, without going back to Specs and you will be able to reproduce it. Also something else I noticed is that regardless what I set for "retries", a failing test is always being retried 4 times.

[cliedeman]

I am also experience this issue.

In my case this was on a working install that hasn't been touched in months

cypress/browsers:node12.8.1-chrome80-ff72

Is there any other external change this could be caused by?

Edit: I managed to resolve my issue after upgrading cypress and the browser image

[emreSmrt]

I am getting the same error very often and it is becoming unbearable

cypress v10.6.0 chrome v104 electron v102

[tusharbmc]

Hi There, Any update on this issue? We are also facing the same issue for lengthy E2E tests which takes around 8-9 mins to execute. I am afraid if Cypress can withstand our complete suite of +1500 E2E tests which we are planning to migrate from Protractor.

Suggestions are welcomed, how to increase Chrome/Cypress memory so that it will not crash. Shall we go headless? will it fix the problem?

[ComeOnGroupAutomationTeam]

Same here, we put a lot of effort to migrate to Version 10, and now it seems like we'll put another effort to revert back to version 9.

[tusharbmc]

Hi @ComeOnGroupAutomationTeam , does that mean Cypress V9 doesn't have this problem even with lengthy tests of duration more than 10 mins?

[ComeOnGroupAutomationTeam]

Hi @ComeOnGroupAutomationTeam , does that mean Cypress V9 doesn't have this problem even with lengthy tests of duration more than 10 mins?

We started to encounter this problem after we upgraded to the new version. We were not encountering this in version 9.4.0.

[emilyrohrbough]

Chrome itself has had quite a few bugs logged around issues with crashing due to running out of memory, so some of the issue is out of Cypress' control. Please see #23391. It seems Chrome 106 should see improvements.

That being said, we understand your frustrations. I am going to do some profiling of Cypress and try to see what we can do on our side to reduce hitting this limit.

Our First steps:

• I am aware of a memory leak introduced in 10.0 that I am actively working on fix.
• I am going to look at destroying our source mapping instances when we tear down Cypress between new spec selections in open mode and in navigation events with cy.visit() to a new domain

Feedback would be appreciated: In addition to altering many tests are kept in memory with numTestsKeptInMemory (default is 50), we know the reporter does impacts the overall performance and large specs will have degraded performance as the run executes due to the number of elements rendered on the dom. We should really virtualize the reporter, however in the short-term, can you try using CYPRESS_NO_COMMAND_LOG=1 (https://docs.cypress.io/guides/references/troubleshooting#Disable-the-Command-Log) to see if reduces/prevents the crashes you are seeing and let us know?

[tusharbmc]

Thanks @emilyrohrbough for your suggestions, surely most of the time numTestsKeptInMemory alteration is not working for us, I will try out NO_COMMAND_LOG along with it and let you know if that helps. I had went through the #23391 you shared; however, I felt like those are the possibilities they talked about and not a confirm fix. What if chrome 106 improvement doesn't fix 'aw snap' error.

I really appreciate your efforts on Cypress profiling and looking forward to it.

[extracker-ppetrov]

Using Electron as a browser and running the tests headless is not reproducing the issue for me. Using Chrome and running the tests headless reproduces the issue. It reproduces most often when using the Runner. More often when using Chrome and less often when using Electron.

[Fewwy]

"cypress": "^9.6.1" - have the same issue using Chrome.

[AnushaAshokReddy]

@jennifer-shehane ,

Is there a final solution on this issue? We are facing same issue for longer test cases even after doing the following settings in cypress.json file

"requestTimeout" : 30000, "numTestsKeptInMemory": 0, "responseTimeout" : 50000, "pageLoadTimeout": 100000,

Could you please post the solution here since there are lot many tickets open on the same issue?

Thank you :)

[extracker-ppetrov]

I am able to reproduce this on Edge and Electron too, for longer test suites that hold multiple test cases, it(}. The issue reproduces mostly when test cases are failing within the spec test suit and are being rerun. Lets say you have 7 test cases in the spec file, if two fail and are rerun 3 times each, the issue reproduces. Setting the numTestsKeptInMemory to something different than 0 is reducing the time to reproduce it.

[emilyrohrbough]

@extracker-ppetrov Thanks for the details. How many tests on average are in your spec? How many commands on average per test? Are you using Cypress out of the box are are you also using a plugin/preprocessor as well?

[extracker-ppetrov]

I would say 8 to 10 tests. My longest test has about 150 lines, but I am using page object model and two layers of encapsulation, so lets say may be 200-250 commands for that one. But that is a rare case. This spec always reproduces the issue when one of the 11 tests inside fails. Mostly I would say my tests have about 50 -60 commands and there are 8 to 10 tests in each spec. Our tests are in TypeScript.
I do not have a preprocessor in my configuration. I am using Cypress out of the box in that aspect.

[extracker-ppetrov]

Here is also a crash output from a headless run. I am not running this in Docker. I am simply using the Terminal in VS Code. I am getting this error for chromium crashing, when I run my tests headless with Electron.
─────────────────────────────────────────────────────────────────────────────── We detected that the Chromium Renderer process just crashed.

This is the equivalent to seeing the 'sad face' when Chrome dies.

This can happen for a number of different reasons:

• You wrote an endless loop and you must fix your own code
• There is a memory leak in Cypress (unlikely but possible)
• You are running Docker (there is an easy fix for this: see link below)
• You are running lots of tests on a memory intense application
• You are running in a memory starved VM environment
• There are problems with your GPU / GPU drivers
• There are browser bugs in Chromium

You can learn more including how to fix Docker here:

https://on.cypress.io/renderer-process-crashed

[Jaroslav004]

I think I had previously version 8 something on my PC with 8GB RAM and around 16 tests when my browser went "Aw snap" on approximately every fourth attempt to run all the tests. I thought that the PC's memory wasn't big enough so I got a new PC with much faster CPU and twice the original RAM and guess what...my tests are now failing on every single attempt. The only difference was that I intalled cypress 10 something instead of version 8 on new PC. It is so bad now that often I'm not even able to run one single (longer) test that ran just fine on version 8.

[extracker-ppetrov]

10.8.0 made things worse. I can hardly run 5-6 tests headless before cypress crashes. I had to revert to 10.7.0.

[tusharbmc]

Chrome itself has had quite a few bugs logged around issues with crashing due to running out of memory, so some of the issue is out of Cypress' control. Please see #23391. It seems Chrome 106 should see improvements.

That being said, we understand your frustrations. I am going to do some profiling of Cypress and try to see what we can do on our side to reduce hitting this limit.

Our First steps:

• I am aware of a memory leak introduced in 10.0 that I am actively working on fix.
• I am going to look at destroying our source mapping instances when we tear down Cypress between new spec selections in open mode and in navigation events with cy.visit() to a new domain

Feedback would be appreciated: In addition to altering many tests are kept in memory with numTestsKeptInMemory (default is 50), we know the reporter does impacts the overall performance and large specs will have degraded performance as the run executes due to the number of elements rendered on the dom. We should really virtualize the reporter, however in the short-term, can you try using CYPRESS_NO_COMMAND_LOG=1 (https://docs.cypress.io/guides/references/troubleshooting#Disable-the-Command-Log) to see if reduces/prevents the crashes you are seeing and let us know?

Hi @emilyrohrbough,

I have updated Chrome 106 and Cypress 10.8, still facing the same issue. Cypress needs some performance improvement to get rid of aw snap error.

Cypress: 10.8, Chrome: 106, already using, numTestsKeptInMemory: 0, Command: npx cypress run NO_COMMAND_LOG=1 --browser chrome --headed --spec */regression/ --config baseUrl=<Application URL>

I am sorry, could not share the code/repo its organization proprietary.

If we are looking at Cypress as a replacement of Protractor, it needs to be robust and reliable. We are not in need of enhanced debug mode or jazzy reports, Cypress should first at least execute our e2e tests which are quite lengthy, some tests take around 20 mins.

I hope Cypress profiling is going well, you and your team will nail down this issue soon.

Thank you!

[vasanthiraman]

Cypress browser is crashing frequently during the execution, it becomes a blank screen suddenly and we need to close the browser window and start the execution again. It is being a major issue since we face it very frequently, especially after the upgrade to 10.6. previously we were using version 9.5 we faced that crash very rarely only when the execution time exceeds 6 or 7 mins. but in the upgraded version we face it even after 3 mins.

We use Electron V102. Please let us know if there is any solution for this issue.