Ensure consistent Chrome versions during CI-parallelization

[muratkeremozcan]

Current behavior

Here's a very ambiguous CI issue during parallelization

It happens because Chrome rolls out varying versions , and if the parallelized CI machines do not match versions, you get the error.

Sporadic issue. It's the first time I saw it externally.

https://github.com/muratkeremozcan/tour-of-heroes-react-cypress-ts/actions/runs/3642676076/jobs/6150068190

You passed the --parallel flag, but we do not parallelize tests across different environments.

This machine is sending different environment parameters than the first machine that started this parallel run.

The existing run is: https://cloud.cypress.io/projects/7mypio/runs

In order to run in parallel mode each machine must send identical environment parameters such as:

 - specs
 - osName
 - osVersion
 - browserName
 - browserVersion (major)

This machine sent the following parameters:

{
  "osName": "linux",
  "osVersion": "Ubuntu - ",
  "browserName": "Chrome",
  "browserVersion": "107.0.5304.121",

To work around it, we have to ensure the same Chrome version in all CI machines

https://github.com/muratkeremozcan/tour-of-heroes-react-cypress-ts/pull/187/commits/f0d37f6fd9bea8d124a30135d2193a8d6c04abc0

      - name: Install Specific Chrome Version
        run: |
          sudo apt-get install -y wget
          sudo wget -q https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
          sudo apt-get install ./google-chrome-stable_current_amd64.deb

This adds CI minutes. It would be ideal if Cypress had a solution to it for parallelization.

Desired behavior

It would be ideal if Cypress Cloud, or the GHA handled parallelization and ensuring consistent Chrome versions between CI machines

Test code to reproduce

See links in description. You will not reproduce this one with code. It will happen sporadically the more parallel CI machines you have.

Cypress Version

12.0.1

Node version

lts - whatever Cypress GHA uses

Operating System

CI - ubuntu

Debug Logs

See description.

Other

No response

[muratkeremozcan]

From @conversaShawn

"Thanks for reporting Murat. The current workaround is to use a specific version as you did or use one of Cypress images: https://github.com/cypress-io/cypress-docker-images/tree/master/browsers Currently, this typically happens whenever a new browser version is released since it grabs the latest. I've also seen the happen when using ubuntu-latest 😢 The good news is, we are actively investigating, so I'll loop in your CSM to keep you in the loop of any updates"

[conversayShawn]

Related to cypress/cypress issues:

• #7199

Related to cypress/github-action issues:

• 121
• 602

Existing workaround:

• Comment

[a28jiang]

Would ideally like to avoid the current workaround as it adds additional overhead to our job duration. It would be great ifparallel-group-params-mismatch could be made less strict!

[essenmitsosse]

Since Chrome 111 was just released, I am encountering this error again. I also agree that I'd rather have parallel-group-params-mismatch less strict if there isn't a more elegant version to ensure all CIs use the same browser. But this needs to be solved. Every few weeks, Cypress tests get super brittle.

If the solution is manually downloading a pinned image, which would add significant time to CI, I'd much rather turn off parallelization and save money on Cypress Cloud. What I want to say: This bug is actively hurting your paid product and making it partially useless.

[jmtimko5]

We are also see this problem. Especially since default versions of browsers on Github runners are not under our control, it would be great if via configurability we could say certain parameters are no of concern.

[arya]

We’re also running into this and don’t want to add significant time to our CI and don’t see a viable workaround. Is there one?

Maybe one option for the cypress team is to add a flag (or just always) to change the behavior to warn on browser version differences. The warning can be close to the test errors, so that if browser versions cause test failures it can be apparent why.

[essenmitsosse]

@ayra: Funny enough, automatic updating (which could cause test errors) isn't even the issue here — it's only when they are out of sync because some workers auto-updated and others didn't. So most users wouldn't even want to be warned about version mismatched. If a new version actually causes a test failure, you are dealing with a whole different can of worms.

[arya]

What’s the reason to not attempt running the tests when the versions are out of sync? It’s not clear why not trying to run the tests when one is on Chrome 110 and one is one Chrome 111

[nirmpate]

In the end we decided to switch our tests to run on electron for the short term just to unblock the team from merging PR's. This worked fine and didn't slow down our pipeline. We're thinking of switching back to Chrome after a couple days. The cypress team should find a real solution to this or else we're stilling going to be getting the same problem once v112 comes out.

[essenmitsosse]

Chrome 112 is out, and it is happening again. Very disappointed about the radio silence because this will cause us a lot of hassle again, and parallelization is the only reason we are paying for Cypress. If this keeps breaking every few weeks, we'd rather switch to the free version and turn off parallelization.

I want to point out this is a core feature of the paid product, and it reliably breaks once a month for like a week.

[MikeMcC399]

Adding one of the cypress/browsers Docker containers to the workflow should work around the issue that GitHub is jumping backwards and forwards in its use of image versions in the same job and different matrix containers during a rollout period.

Docker lists the available containers on https://hub.docker.com/r/cypress/browsers/tags

The GitHub Action CI documentation for Parallelization shows the use of a Docker container, although the current example container: cypress/browsers:node12.18.3-chrome87-ff82 is outdated.

One example would be:

jobs:
  install:
    runs-on: ubuntu-22.04
    container: cypress/browsers:node18.12.0-chrome107

otherwise there are newer images which have Chrome, Firefox and Edge included.

Edit: GitHub Actions product feedback added in https://github.com/orgs/community/discussions/52956.

[MikeMcC399]

cypress-io/github-action > README > Parallel now contains the specific advice:

"During staged rollout of a new GitHub-hosted runner version, GitHub may provide a mixture of current and new image versions used by the container matrix. It is recommended to use a Docker image in the parallel job run which avoids any Cypress Cloud errors due to browser major version mismatch from the two different image versions. A Docker image is not necessary if testing against the default built-in Electron browser because this browser version is fixed by the Cypress version in use and it is unaffected by any GitHub runner image rollout."

[MikeMcC399]

Cypress documentation Specifying Browsers in Parallel Builds now describes precautions to prevent this issue causing errors:

GitHub runners are currently preparing to update from Google Chrome 112.0.5615.165 to 113.0.5672.63.

GitHub product management didn't respond so far to https://github.com/orgs/community/discussions/52956

[MikeMcC399]

The workaround of using a Cypress Docker image is only available for Ubuntu / Linux on GitHub. These are the only operating systems supported as containers on GitHub. Windows and macOS are not supported. See https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idcontainer

[MikeMcC399]

• According to the feedback in the GitHub actions/runner issue https://github.com/actions/runner/issues/2812#issuecomment-1708710610 no change can be expected in the way that GitHub Actions Backend randomly assigns image versions in response to requests for a virtual machine. So the current best advice is still to use a Docker image or otherwise install fixed browser versions when recording to Cypress Cloud.

[nick-fields]

Our app requires MacOS and bare metal GPU access, so the docker images are not helpful here. What compounds the issue is Chrome's rolling updates, and how they don't really support disabling auto updates (the recommended Keystone plist updates seem to be no longer supported).

Can the check for browserVersion at least be made configurable?

[a8trejo]

Don't know why i don't see it highlighted here, but this workaround has worked for me @nick-fields https://github.com/cypress-io/github-action/issues/518#issuecomment-1210979047

[bmulholland]

I'm hitting this error now and it just seems so strange. Cypress has intentionally added a conditional that crashes some workers during a run, which breaks builds... but why? Has anyone received an explanation for why this is the case? I'm fine with Chrome being different between different workers on the same runs. This effectively breaks the parallel test runs that we pay for with Cypress, doesn't it? Am I missing something?

I believe any workaround for this requires us to manually manage Chrome versions, monitoring Chrome updates and changing a version number manually, and/or additional CI minutes to install versions. Is there one that doesn't cost me CI time and doesn't require us to add yet another version of something to manage and upgrade over time?

[jennifer-shehane]

The logic errors when there is a difference in MAJOR browser version between two specs within the same parallelized run. A core component of testing and getting reliable results is ensuring a stable environment in which all of the tests are run. If one specfile runs in Chrome 119 and one runs in Chrome 120, you may get mixed results due to the vast amount of changes that Chrome makes in major updates. We encountered this ourselves - our own internal tests at Cypress began failing with a Chrome update when they changed their behavior around http to https upgrades. You would end up with mixed results in this situation.

I would encourage you to look towards solutions that maintain a stable browser version to run your tests within - using our provided docker-images or other solutions that ensure browsers don't auto-update in the middle of runs.

We'll leave this issue open for future consideration, but to accomplish this would be Cypress providing our own browsers and a versioning system which we spent some time investigating but is a lot of work to implement.

[bmulholland]

Thanks for the response, @jennifer-shehane. I understand that upgrades like that may cause inconsistencies in results -- and I'm happy to take on the responsibility for that. I'd rather it over Cypress failing my test suite entirely, or having to manage Chrome versions manually. Why doesn't Cypress give me an option to "enable browser inconsistencies" or something? Opt-in, with a clear warning, and just disable this one check.