10.3.0 regression - test runner randomly fails when running electron locally

[glomotion]

Current behavior

So when running COMPONENT tests specs locally, the test runner randomly crashes out at varying specs. 🤔

ERR_FAILED (-2) loading 'http://localhost:51860/__/#/specs/runner?file=src/components/section-heading/section-heading.test.tsx'
Error: ERR_FAILED (-2) loading 'http://localhost:51860/__/#/specs/runner?file=src/components/section-heading/section-heading.test.tsx'
    at rejectAndCleanup (node:electron/js2c/browser_init:161:7647)
    at EventEmitter.stopLoadingListener (node:electron/js2c/browser_init:161:8022)
    at EventEmitter.emit (node:events:390:28)
    at EventEmitter.emit (node:domain:475:12)

Sometimes it fails at other specs. I have tested every single spec where it has crashed using cypress open - and all specs pass when run manually. I think there's something up with the test runner inside of MACOS 12.4 (21F79)

Sometimes the test runner gets as far as (7 of 64) sometimes (16 of 64), and sometimes it gets to (26 of 64). 🤔 Im having a tough time trying to work out why the runner crashes out at various different specs, there is almost nothing else running while the cypress runner is working.

Desired behaviour

for all the specs to run, just like they do in CI

Test code to reproduce

Ill try and update this ticket with these details, but was hoping to gauge how many people are also finding this issue right now

Cypress Version

10.3.0

Other

Some other relevant dependencies:

"@emotion/css": "^11.5.0",
"react": "^17.0.2",
"react-dom": "^17.0.2"
"cypress-real-events": "^1.7.1",
"@cypress/code-coverage": "^3.10.0",

MacOS 12.4 (21F79) running on an M1 MAX

[glomotion]

whats very strange, is that running the tests with coverage turned on, the test runner gets through all tests, without crashing. When i disable code coverage, that when the test runner randomly crashes out. So weird.

[lmiller1990]

Hi Tim - thanks for the bug report, your issues are always informative.

Normally the issue is the opposite - CI fails, local passes. Disabling code coverage -> success is even more weird. What plugins/libs/versions are you using for code coverage?

Does this happen in the previous version of Cypress? If it really was introduced 10.3.0, that's a small surface area, it should be easy to fix - can you provide a minimal reproduction, along with answers to the above questions?

[glomotion]

Hey again Lachlan :) (and awww thank you!)

OK so to try and answer your questions:

• It now seems I was mistaken about the test runner NOT crashing locally with coverage enabled - it just crashed on me then while coverage was enabled.
• FYI we use "@cypress/code-coverage": "^3.10.0", (i'll add that dep to the OP above).
• This definitely did NOT happen when we were on 9.6.x - the local test runner was bullet proof.

As for the repro, that will take some time (the repo in question is private, sadly 😢). I'll see if i can throw something together in the next day or so and report back.

[glomotion]

So i've been trying to create a stripped down version of the repo in question - to demonstrate the error happening - but it seems that with less specs - i cannot get the issue to happen. :(

And I don't think i can show you a demo where the issue is happening because id have to move a whole private design system repository into a public repo - just to show the error happening.

Whilst trying to provide the repro - i've tried deleting any specs which cause the error - 1 by 1 - but each time it will randomly crash on a new spec. Again, each of the specs that it crashes out on, actually pass when i run them manually.

Something which is worth noting: whenever i run the cypress test runner locally (in mac os) after upgrading to 10.x - i get the following error log at the start:

objc[48773]: Class WebSwapCGLLayer is implemented in both /System/Library/Frameworks/WebKit.framework/Versions/A/Frameworks/WebCore.framework/Versions/A/Frameworks/libANGLE-shared.dylib (0x253f4f808) and /Users/tim/Library/Caches/Cypress/10.3.0/Cypress.app/Contents/Frameworks/Electron Framework.framework/Versions/A/Libraries/libGLESv2.dylib (0x109a313c8). One of the two will be used. Which one is undefined.

I overlooked that above error before, because the test runner seems to continue on afterwards - but maybe theres more to this error output than i first clocked.

Ive also tried upgrading all the components to use react 18.x - this also does not help.

Honestly it feels like it's some kind of weird memory related issues, because it doesn't seem to start happening until there are lots of specs (~64 or so).

[glomotion]

hey there again @lmiller1990! So i had an idea overnight, and have actually managed to strip down the repo to just a few components and tests, then duplicated everything 3 or 4 times over ... so that this repo now replicates the test runner issue we're seeing locally on our end 🎉

https://github.com/glomotion/test-cypress-typescript-react/

when you run npm t you should see the test runner crash out at a random point like this:

(its a private repo, so only you and I have axs to it right now)

[lmiller1990]

@glomotion Right, that's what the GH repo invite was. (Was out for 1 week).

I will try running this and see if I run into the same problem. Having a minimal repro makes it sooooo much easier to debug and by extension much more likely the problem is solved, thanks for this!

[lmiller1990]

@glomotion are you able to invite @BlueWinds to your repo, too? They will be taking a look at this one. Thanks!

[glomotion]

@BlueWinds & @lmiller1990 done! :)

[BlueWinds]

Just writing down some things I've tried to help narrow down the problem.

• Crashes completely reliably on Kubuntu 22.04, so not mac / M1 related.
• Removing code coverage did not fix it for me - I still get reliable crashes even with that gone.
• --headed: Still crashes.
• --browser chromium / --browser firefox: No crash. (tests fail on firefox, but that's unrelated)
• Cypress 10.0.0: Still crashes.

So in short, it's electron specific, operating system independent, and not related to the 18.0.4 -> 18.3.0 Electron upgrade that happened in Cy 10.2.0 (which was my guess).

I ran the tests with DEBUG=*, but this didn't yield anything useful - electron just straight up dies when we're asking it to load the next spec, no message beyond the one you already posted.

https://github.com/electron/electron/issues/28208#issuecomment-944222413 is the only potentially relevant Electron bug I was able to find - specifically:

As an additional case, this error seems to happen if the page code has an onbeforeunload event handler that blocks the unload. Then the same 'GUEST_VIEW_MANAGER_CALL': Error: ERR_FAILED (-2) happens and the loadURL fails.

sounds like it could be potentially relevant.

@glomotion - What's the last 'known good' Cypress version (if any) where you weren't running into this?

[BlueWinds]

I should also note that I'm seeing this crash super reliably right after the 32nd spec, rather than jumping around to various locations as you described. Could possibly be an OS difference.

[glomotion]

@BlueWinds it seems all versions that we've tried of 9.x (eg 9.6.1) are totally fine. We only spotted this issue after migrating one of our codebases to cypress 10.x

[lmiller1990]

@BlueWinds that's interesting, glad we can at least reproduce the "crashes after X specs" bug. There is someone else having the same issue (after 138 specs): https://github.com/cypress-io/cypress/issues/22208

They reported better results w/ Chrome. Either way these issues look suspiciously similar.

[BlueWinds]

Yeah, it appears to work fine under Chrome, no crashes at all, so that's a workaround. Added 'electron' to the title to help others locate the issue.

[glomotion]

FYI i worked out that the tests only ran in CI because we use chrome in CI. 🤪 I have updated the title to remove the reference to it being OK in CI

[lmiller1990]

Thanks! Good to know this is isolated to electron... why is electron less reliable, I thought it would be shipping pretty much the same code as chrome, at least the core components.

[wilsonhou]

+1 to this! would appreciate this fixed 🥲

[danny-faith]

+1 ! Love to see a fix for this

[lmiller1990]

I think we'd all love to see this fixed :smiling_face_with_tear: along with this other long standing electron issue: https://github.com/cypress-io/cypress/issues/15932

I am really unsure if this is even a bug in Cypress or a bug in electron. If anyone wants to dig into this, happy to assist - tons of good info here, but I don't think I can actively debug this in the immediate future.

[glomotion]

@lmiller1990 i wonder, is it possible to downgrade cypress 10.x's electron dependency to a previous version where this issue does not happen? This particular issue isn't present at all for cypress < 10.x. 🤔

We're desperate to upgrade to cypress 10.x so that we can finally upgrade to nextjs 12.x - and this is issue is the one thing that's blocking us atm (there are those of us on the team who refuse to use chrome - on the count of the horrendous keystone perf issue). 😢

[lmiller1990]

Hmm good point, this got me thinking... it's definitely possible to use a different version of Electron if you have it installed locally. Documentation.

We should be able to isolate this. First, we need to answer the question:

• is this an Electron version issue (eg - would downgrading electron even solve the issue)?
• or, is this a Cypress 10 only issue?
• finally, does this only happen with component testing? That is the assumption here - I didn't think about it too much, but if this is a CT only issue, that narrows the surface area a lot.

Downgrading the electron version we ship is a little more involved in just changing a dependency, but maybe we debug it like this:

1. Run Cypress 10 and passing different electron versions, like documented: cypress run --component --browser /usr/bin/some-electron-version
2. Try lots of Electrons
3. Try the same thing, but with Cypress 9
4. Isolate issue!

Good input - I feel like we are slowly closing in on this.

[glomotion]

Ohhh I think those docs have kindof unblocked us for cypress 10.x 🎉 Now i can run tests locally using brave, which is all i really need to move fwd!! :)

If you offer some instructions for how to install electron locally at a specific version (having trouble working out how this is possible atm 🤔), I can also give that a whirl to try and help you narrow down the issues

[lmiller1990]

kindof unblocked us for cypress 10.x

Great! Not entirely clear how this unblocks you, but glad it does.

electron versions

Hm, for Electron versions, can we just do npm install -g electron@<version> to install the version you want, then point at the binary (where-ever that installs, probably depends on your package manager and OS)?

[glomotion]

Great! Not entirely clear how this unblocks you, but glad it does.

yeah - so i didn't realise it was possible to use brave to run cypress tests. This means we can run tests without failure locally - which was what was blocking us adopting cypress 10.x

Hm, for Electron versions, can we just do npm install -g electron@ to install the version you want, then point at the binary (where-ever that installs, probably depends on your package manager and OS)?

So electron versions seem to be different in NPM from how they're used in cypress. 🤔 https://www.npmjs.com/package/electron has the latest version as 20.0.1 - but the version described inside cypress test runs is 104.

[lmiller1990]

Yeah it's confusing - the major version on npm is the major version of the electron binary, which is different to the chomium binary version.

I am not deeply familiar with their update strategy but you can figure out which npm versions ship which chomium binary by looking here: https://github.com/electron/electron/blame/032e1d9befad640ee6edc276cec1ebd1e84bc73f/DEPS#L5

In fact I think we can just bypass installing electron and download a specific Chromium version. I will try now and report back.

[lmiller1990]

Ok, I looked into this, here's how you can use old Chromium versions. First, obtain the version you want. I grabbed one here: https://vikyd.github.io/download-chromium-history-version/#/

I downloaded the zip called chrome-linux.zip. I also noticed Cypress has some old Chrome versions, if that's useful to someone. Those are here.

I got my desired version (say 90) and extracted it to my desktop:

$ ls /home/lachlan/Desktop/chr/chrome-linux

chrome			icudtl.dat	       nacl_irt_x86_64.nexe
chrome_100_percent.pak	libEGL.so	       product_logo_48.png
chrome_200_percent.pak	libGLESv2.so	       resources
chrome_sandbox		locales		       resources.pak
chrome-wrapper		MEIPreload	       swiftshader
ClearKeyCdm		nacl_helper	       v8_context_snapshot.bin
crashpad_handler	nacl_helper_bootstrap  xdg-mime
extensions		nacl_helper_nonsfi     xdg-settings

chrome is the binary. I'm on linux so I can just ./chrome to launch it, if you want to try browsing the web with an old verison of Chromium for whatever reason.

You can customize available browsers in Cypress. See the docs. I did it:

import { defineConfig } from 'cypress'

export default defineConfig({
  'e2e': {
    'setupNodeEvents': (on, config) => {
      config.browsers = config.browsers.concat({
        name: 'Old Chromium',
        channel: 'stable',
        family: 'chromium',
        displayName: 'Old Chromium',
        version: '90',
        path: '/home/lachlan/Desktop/chr/chrome-linux/chrome', // <= this is the important part, pass the path
        majorVersion: '90',
      })
      return config
    },
    'baseUrl': 'http://localhost:3500',
  },
})

Now I can see it:

You could easily queue up many versions like this for testing. I also think you can just do yarn cypress open --browser <path_to_browser> but I like to have lots of them available for testing various things.

[glomotion]

Alrighty! thats awesome intel, thank you @lmiller1990 - ill try and give that a go and see if the error still happens - and report back. :)

[chenxsan]

Hmm good point, this got me thinking... it's definitely possible to use a different version of Electron if you have it installed locally. Documentation.

We should be able to isolate this. First, we need to answer the question:

• is this an Electron version issue (eg - would downgrading electron even solve the issue)?
• or, is this a Cypress 10 only issue?
• finally, does this only happen with component testing? That is the assumption here - I didn't think about it too much, but if this is a CT only issue, that narrows the surface area a lot.

Downgrading the electron version we ship is a little more involved in just changing a dependency, but maybe we debug it like this:

1. Run Cypress 10 and passing different electron versions, like documented: cypress run --component --browser /usr/bin/some-electron-version
2. Try lots of Electrons
3. Try the same thing, but with Cypress 9
4. Isolate issue!

Good input - I feel like we are slowly closing in on this.

We ran into a similar problem when upgrading cypress 10.7.0 to 10.8.0, where tests are not component testing. Using chrome browser could work around the problem, see https://github.com/webpack/webpack.js.org/pull/6415. But I'm not quite sure that it's electron related since cypress has same electron version in both 10.7.0 and 10.8.0, and we didn't run into any issue with 10.7.0 before.

[lmiller1990]

@chenxsan did you see CI fail multiple times or just once? I wonder if it was just flake - I posted in your PR with the dif from 10.7->10.8 but I don't see any obvious thing that would impact your tests.

Can you reproduce locally or only on CI?

[chenxsan]

@chenxsan did you see CI fail multiple times or just once? I wonder if it was just flake - I posted in your PR with the dif from 10.7->10.8 but I don't see any obvious thing that would impact your tests.

Can you reproduce locally or only on CI?

I've tried that failed CI many times, all of them failed.

Yeah, that's why I was surprised that using chrome wouldn't throw any errors.

I have some network problems here when fetching some public api so I can't run the code locally.

[lmiller1990]

@chenxsan i think thien is related https://github.com/cypress-io/cypress/issues/23830