Cypress crashes with "We detected that the Chromium Renderer process just crashed."

[Lasidar]

Test code to reproduce

I cannot provide the specific test code I am using to produce this, since it contains internal company data. However, what I can tell you is these tests run for quite a long time (5-10 minutes), as they contain a lot of steps through our E2E flow in our application.

Cypress Mode

cypress run

Cypress Version

12.13.0

Browser Version

Chrome 110.0.5481.96

Node version

v16.18.1

Operating System

Debian GNU/Linux 11 (Docker image cypress/browsers:node-16.18.1-chrome-110.0.5481.96-1-ff-109.0-edge-110.0.1587.41-1)

Memory Debug Logs

Only the end of the failing test results from the memory dump have been included due to limitations of this input form.

[
  {
    "checkMemoryPressureDuration": 7764.029118001461,
    "testTitle": "Crashing test (example #1)",
    "testOrder": 18.2,
    "garbageCollected": true,
    "timestamp": 1690553650738
  },
  ...
  {
    "getRendererMemoryUsageDuration": 2.619260013103485,
    "totalMemoryWorkingSetUsed": 6995816448,
    "getAvailableMemoryDuration": 58.69076597690582,
    "jsHeapSizeLimit": 4294705152,
    "totalMemoryLimit": 9223372036854772000,
    "rendererProcessMemRss": 5469155328,
    "rendererUsagePercentage": 127.34646813770371,
    "rendererMemoryThreshold": 2147352576,
    "currentAvailableMemory": 9223372029858955000,
    "maxAvailableRendererMemory": 4294705152,
    "shouldCollectGarbage": true,
    "timestamp": 1690553801030,
    "calculateMemoryStatsDuration": 58.72436600923538
  },
  {
    "getRendererMemoryUsageDuration": 2.208419978618622,
    "totalMemoryWorkingSetUsed": 5089853440,
    "getAvailableMemoryDuration": 61.31387501955032,
    "jsHeapSizeLimit": 4294705152,
    "totalMemoryLimit": 9223372036854772000,
    "rendererProcessMemRss": 0,
    "rendererUsagePercentage": 0,
    "rendererMemoryThreshold": 2147352576,
    "currentAvailableMemory": 9223372031764918000,
    "maxAvailableRendererMemory": 4294705152,
    "shouldCollectGarbage": false,
    "timestamp": 1690553802092,
    "calculateMemoryStatsDuration": 61.33369600772858
  },
  {
    "getRendererMemoryUsageDuration": 2.69021999835968,
    "totalMemoryWorkingSetUsed": 1682976768,
    "getAvailableMemoryDuration": 50.05962598323822,
    "jsHeapSizeLimit": 4294705152,
    "totalMemoryLimit": 9223372036854772000,
    "rendererProcessMemRss": 0,
    "rendererUsagePercentage": 0,
    "rendererMemoryThreshold": 2147352576,
    "currentAvailableMemory": 9223372035171795000,
    "maxAvailableRendererMemory": 4294705152,
    "shouldCollectGarbage": false,
    "timestamp": 1690553803143,
    "calculateMemoryStatsDuration": 50.07922601699829
  },
  {
    "getRendererMemoryUsageDuration": 2.889739990234375,
    "totalMemoryWorkingSetUsed": 1682792448,
    "getAvailableMemoryDuration": 60.31445497274399,
    "jsHeapSizeLimit": 4294705152,
    "totalMemoryLimit": 9223372036854772000,
    "rendererProcessMemRss": 0,
    "rendererUsagePercentage": 0,
    "rendererMemoryThreshold": 2147352576,
    "currentAvailableMemory": 9223372035171979000,
    "maxAvailableRendererMemory": 4294705152,
    "shouldCollectGarbage": false,
    "timestamp": 1690553804204,
    "calculateMemoryStatsDuration": 60.33361500501633
  },
  {
    "getRendererMemoryUsageDuration": 2.6974300146102905,
    "totalMemoryWorkingSetUsed": 1682558976,
    "getAvailableMemoryDuration": 225.94400304555893,
    "jsHeapSizeLimit": 4294705152,
    "totalMemoryLimit": 9223372036854772000,
    "rendererProcessMemRss": 0,
    "rendererUsagePercentage": 0,
    "rendererMemoryThreshold": 2147352576,
    "currentAvailableMemory": 9223372035172213000,
    "maxAvailableRendererMemory": 4294705152,
    "shouldCollectGarbage": false,
    "timestamp": 1690553805431,
    "calculateMemoryStatsDuration": 225.9711429476738
  }
]

Other

Our test specs that contain multiple long running tests are prone to crashing mid-run in CI. This seems to be more likely when there are test retries in the run. We are running with both experimentalMemoryManagement set to true and numTestsKeptInMemory set to 0. We also have the memory and CPU allocation in our GitLab runners set quite high (see below). Despite this, we still get the crashes. Example:

Some top level test description
    (Attempt 1 of 4) A test scenario containing a scenario outline template (example #1)
    (Attempt 2 of 4) A test scenario containing a scenario outline template (example #1)
    ✓ A test scenario containing a scenario outline template (example #1) (849857ms)
    ✓ A test scenario containing a scenario outline template (example #2) (360954ms)
    ✓ A test scenario containing a scenario outline template (example #3) (556574ms)
    (Attempt 1 of 4) A test scenario containing a scenario outline template (example #4)

We detected that the Chromium Renderer process just crashed.

This can happen for a number of different reasons.

If you're running lots of tests on a memory intense application.
  - Try increasing the CPU/memory on the machine you're running on.
  - Try enabling experimentalMemoryManagement in your config file.
  - Try lowering numTestsKeptInMemory in your config file during 'cypress open'.

You can learn more here:

https://on.cypress.io/renderer-process-crashed

Here are the memory allocations we are providing in Gitlab CI:

  KUBERNETES_CPU_REQUEST: "3"
  KUBERNETES_CPU_LIMIT: "4"
  KUBERNETES_MEMORY_REQUEST: "12Gi"
  KUBERNETES_MEMORY_LIMIT: "32Gi"

It should be noted these tests run within docker in CI, and are running in the cypress/browsers:node-16.18.1-chrome-110.0.5481.96-1-ff-109.0-edge-110.0.1587.41-1 version of the cypress image.

We are utilizing the cypress-cucumber-preprocessor library, but I do not believe that is having any impact on this issue.

[Lasidar]

I have a speculation as to what is causing this which is based on a few logical leaps, but I think they are reasonable. I think the issue is mainly caused by the open issues involving cypress and K8s (see https://github.com/Zenika/alpine-chrome/issues/109).

Looking at the logs in question, I noticed the jsHeapSizeLimit limit is being exceeded right before the crash.

    "jsHeapSizeLimit": 4294705152,
    "rendererProcessMemRss": 5469155328,
    "rendererUsagePercentage": 127.34646813770371,

This is strange, since we have 12GB allocated to the Gitlab runner, with the ability to scale up to 32GB if the process calls for it. But looking at the issue linked above, I believe this prevents the Chrome renderer from being able to scale up its heap memory usage, even if the system has capacity.

From looking at the Cypress source code, it seems the renderer process memory is pulled from the chrome jsHeap, and if we exceed this, we are likely smashing the stack which probably leads to the crash.

  // the max available memory is the minimum of the js heap size limit and
  // the current available memory plus the renderer process memory usage
  const maxAvailableRendererMemory = Math.min(jsHeapSizeLimit, currentAvailableMemory + rendererProcessMemRss)

So my workaround, at least temporarily, is to increase the chrome jsHeapSizeLimit value. I was able to achieve this through the following additions to my cypress/plugins/index.js file:

module.exports = (on, config) => {
  on("before:browser:launch", (browser = {}, launchOptions) => {
    if (browser.family === "chromium") {
      launchOptions.args.push('--js-flags="--max_old_space_size=1024 --max_semi_space_size=1024"');
    }
    return launchOptions;
  });
});

This seems to have made my memory crashing issues go away for the time being. I believe the correct fix for this is for https://github.com/Zenika/alpine-chrome/issues/109 to be resolved, since as I mentioned above, I suspect this issue prevents the process from being able to properly scale up its heap size.

[Pardoniano]

I started to have the same problem this week. Already tried your solution on cypress.config.ts, with --max_old_space_size=4096. And still crash on a specific element. He crashes in Chrome and Edge, but work's fine in electron.

[uladzimirdev]

increasing max_old_space_size helps locally, but I can clearly see memory usage increase on chrome 115 - it can be 3.8Gb in my case

[Pardoniano]

I tested it in the homolog environment, I encountered this error. Interestingly, the button who causes the crash it works perfectly outside of Cypress, and I have tested it in various scenarios with cypress. The version of Cypress used is 12.7.0.

Here are the details of the environments where the issue occurred:

1. Environment: Windows 22621.1992 Browser: Chrome - Version 115.0.5790.110 (Official Build) (64-bit) Result: Didn't work

2. Environment: macOS Ventura version 13.4.1 (c) (22F770820d) Browser: Chrome - Version 115.0.5790.114 (Official Build) (arm64) Result: Didn't work

3. Environment: macOS Ventura version 13.1 (22C65) Browser: Chrome - Version 115.0.5790.114 (Versão oficial) (arm64) Result: Didn't work

4. Environment: Ubuntu Browser: Chrome - Version 113.0.5672.126 Result: Worked fine

5. Also tested it in our pipeline, and it worked without any issues. The Docker image used was cypress/browsers:node16.14.2-slim-chrome100-ff99-edge.

I suspect that the Chrome version 115 might be causing the problem, possibly due to memory leaks or other issues, idk, just guessing so far. Additionally, in the Electrum application, after clicking the specific button he shows in the cypress console one of the following error messages:

1. ResizeObserver loop limit exceeded;
2. ResizeObserver loop completed with undelivered notifications.

Wich i already fixed way back then with Cypress.on("uncaught:exception")

Unfortunately, this is all the information I currently have, as I haven't had enough time to investigate further. And i tried the workaround, sadly didn't work. So far my workaround is gonna be work on electron.

[nagash77]

Hi everyone on this thread. Thank you for submitting this ticket and all the details and investigation you all have done. We do continually revisit performance and resource utilization within Cypress so these are great data points. Unfortunately due to the nature of these problems they can be extremely hard to reproduce and even harder to track down so I cannot give any timelines about a resolution. If anyone seeing this ticket has ideas on culprits, I encourage you to open a PR and we will work with you to get that merged if it shows that it can help this problem.

[Adrian-Sana]

I am having the same issue, it is crashing with Chrome 115 but if I download the stable 114 from here: https://chromium.cypress.io/mac/ and then run it with --browser [path to the downloaded chrome] it works.

[Lasidar]

@nagash77 I understand that actually finding the root cause here will be extremely difficult. One thing I did find interesting when I was analyzing the memory usage leading up to the crash is between tests, the memory usage never returns back down to a baseline level. It is continually increasing. This is even with experimentalMemoryManagement set to true and numTestsKeptInMemory set to 0. I find this strange since I am not sure what would be persisting in memory between tests.

If a true root cause fix is not possible, I have a few possible suggestions:

• Give the users control over the browser heap size allocation via a parameter in cypress.config.js. That way, instead of setting it through a before:browser:launch trigger, it could be set more transparently via a test configuration option.
• In the event of a renderer crash, perhaps we could print more verbose memory statistics (i.e. max rendererProcessMemRss used, jsHeapSizeLimit, etc).
• If possible, try and solve the K8s / docker issue (i.e. https://github.com/Zenika/alpine-chrome/issues/109), which I believe may be contributing to this issue.

[ChristianKlima]

It seems as if Chrome has been overwhelmed with the layout of our application including Cypress since version 115. Our application runs unstably when command-log window is displayed. As soon as an action is performed in the application that writes a new entry in the command log, the browser crashes. This is usually not easy to reproduce, but we were able to reproduce it in a test at some point. Debugging the Chrome crashdumbs didn't help us either. Everything was stable up to version 114 of Chrome. Our solution is to set CYPRESS_NO_COMMAND_LOG: 1 now. Maybe it will help someone. We hope that at some point the problem will solve itself with a Chrome update or with a simpler cypress layout. Unfortunately, this is not that easy to reproduce in a small sample application.

[olahaida]

I ran in to the same issue. Set of test that I was running without any problems a month ago, now is crushing with

"We detected that the Chromium Renderer process just crashed. This can happen for a number of different reasons. If you're running lots of tests on a memory intense application.

• Try increasing the CPU/memory on the machine you're running on.
• Try enabling experimentalMemoryManagement in your config file.
• Try lowering numTestsKeptInMemory in your config file during 'cypress open'."

I've added experimentalMemoryManagement: true to my config file. Also reduced the numTestsKeptInMemory, but it's still crashing. Did not crash only when I reduced the number of tests to 0, but it's not solving my problem because I need to see those results (and a month ago it was working without any reduction).

I can see in the Task Manager that the process is crashing whenever Cypress is reaching over 4GB of RAM, I know it was always consuming a lot of resources but not that much

Windows 10 Cypress version: 12.17.4 Chrome v116 Electron v106

[AMEntwistle]

We also ran into this issue, particularly with chrome v116. We currently are locking to chrome version 115.0.5790.110-1 which is stable as far as we can see in regards to this memory issue. I don't know if this is more an issue with chrome or cypress, but being unable to test on latest browser versions is not ideal.

[testauto05]

I am getting error consistently on Windows machine with all browsers from command line whether headed or not. It only works once in a while in Electron when run from the App. Please suggest any workaround for this.

Chrome, Edge, Electron - all latest versions. Cypress - 12.14 and 12.17

[Lasidar]

As mentioned in the original ticket, we are experiencing this in Chrome 110.0.5481.96, so at least for us, it does not appear to be a case where Chrome 116+ is causing it. However, we are going to try Chrome 115 and report back.

[testauto05]

Thanks for reply. It works fine by increasing experimentalMemoryManagement to 4GB. But more scenarios are to be added to our suite, so even 4GB may not be sufficient later. Cypress team please find better option. @nagash77

[VitaliLebedzeu]

I am getting error consistently on Windows machine with all browsers from command line whether headed or not. It only works once in a while in Electron when run from the App. Please suggest any workaround for this.

Chrome, Edge, Electron - all latest versions. Cypress - 12.14 and 12.17

The same problem. Browser crashed with all browsers. Started in chrome only and switching on the electron helps to solve this issue, but now it happens on all browsers.

[akondratsky]

Our team faced this issue, too. We use Cypress 12.17.4 with Electron, and all tests run in the Docker container (built on top of the CentOS image). We tried suggestions from the article Fixing Cypress errors part 1: chromium out of memory crashes, but none of them helped (we did not check Chromium option).

It is worth noting that issue appears only on one machine. Three of the other works fine, without crashes at all. Same image, being run in our Jenkins pipeline, also works smoothly. We've already checked our Docker configurations - they are the same. All machines except Jenkins agent under the MacOS. And it is obvious that different processors also do not play a crucial role here.

I wonder what can be the difference here. If anyone has any idea what we can check - please ping me. We will return to investigating tomorrow.

[kevingorry]

@akondratsky have you found a solution ? I'm running into something very similar

[valeraine]

After recent Chrome updates (144 onwards) our cypress e2e test keep crashing like crazy, tests that have never had those issues before with 3 to 4 test cases...

[akondratsky]

@kevingorry unfortunately, we did not find a solution. We have tried different options, like disabling GPU / rasterization and some other stuff, but without success.

One idea I have come to is about logging. Per as I understand this topic, here may be hidden not just a single issue, but a few, and one of the problems is that it is impossible to figure out a reason for why Electron is crashing. This stuff is far beyond of my competencies, and I cannot dive deep enough into the topic because of our deadlines.

[kevingorry]

@akondratsky I'm in the same boat, I run my tests on Azure runners (Ubuntu + Electron 106) on Cypress 13.1.0 with 16GB of memory and they still crash that is ridiculous ! I enabled everything they said numTestsKeptInMemory: 0, experimentalMemoryManagement: true,

@nagash77 Any news on this ongoing issue ?

[irbrad]

I, too, am in the same boat on GHA runners. 😞 We're on Chrome 114 running on Cypress 13.2 now. Maybe 50% of the runs are crashing out. I had to override the timeout from the default six hours to ten minutes because of it.

[uladzimirdev]

as a temp solution you can use old version of chrome at GHA

[irbrad]

as a temp solution you can use old version of chrome at GHA

How old? I just backtracked to 108 and the same thing is happening. Going to try going back to Cypress 12.

[irbrad]

It doesn't seem to be crashing on Chrome 109 with Cypress 12.17.4.

[uladzimirdev]

How old? I just backtracked to 108

@irbrad we sticked to the 111 version, it does not crash for us

[j0sephx]

we're experiencing this on chromium 112 & cypress version 13.2.0 - set numTestsKeptInMemory: 0, experimentalMemoryManagement: true - will try changing the version to 111 unless there is a resolution that hasnt been posted?

[dheerajpiplani]

I have started facing this issue. I tried all the solutions but still crashing. Running on pipeline Electron browser Cypress 12.17.1 Does anyone has any valid solution to this?

[irbrad]

I have started facing this issue. I tried all the solutions but still crashing. Running on pipeline Electron browser Cypress 12.17.1 Does anyone has any valid solution to this?

I downgraded to Cypress 12.17.4 and Chrome 109. All seems fine now.

[ornakash]

+1

[niharika-i6-io]

I am having this issue on macOS Ventura, Chrome Version 117.0.5938.62 (Official Build) (arm64)

[hammzj]

Experiencing this quite often. We also use the cypress-cucumber-preprocessor.

Docker image is built from cypress/browsers:node-18.16.0-chrome-114.0.5735.133-1-ff-114.0.2-edge-114.0.1823.51-1.

Node version: 18.16.0

Chrome version: 114.0

Cypress version: 12.12.0

---

Tried with numTestsKeptInMemory: 1 and experimentalMemoryManagement: true and it would occur, especially when there would be a test failure with retries. It seems that once failures compound, then there is a higher chance of the Chromium renderer crashing.

So far, the only solution I've found is to reduce the amount of tests per feature (spec) file and split them out amongst multiple feature files. It seems that each new spec file will clear out any problems with memory, but that's patching an issue that feels to have been existing since Cypress 12 has been released.