Cypress freezes in Docker when running in Firefox

[mj-jackson]

Hi, I was using Cypress 7.0.0 and the docker image cypress/included was working. After updating to 7.1.0 the docker container started to "freeze" on random tests. I tried to downgrade to 7.0.1 = didn't work, downgrading to 7.0.0 made it work again.

I say it "freezed" because there are no outputs to the console anymore and the container never finishes.

After running it with DEBUG=cypress:* the outputs stop at some point. Every Minute or so it updates and shows "current & mean memory and CPU usage by process group" looking a little like this:

cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 56106 } +22s
  cypress:server:util:process_profiler current & mean memory and CPU usage by process group:
  cypress:server:util:process_profiler ┌─────────┬───────────────────┬──────────────┬────────────────────┬────────────┬────────────────┬──────────┬──────────────┬─────────────┐
  cypress:server:util:process_profiler │ (index) │       group       │ processCount │        pids        │ cpuPercent │ meanCpuPercent │ memRssMb │ meanMemRssMb │ maxMemRssMb │
  cypress:server:util:process_profiler ├─────────┼───────────────────┼──────────────┼────────────────────┼────────────┼────────────────┼──────────┼──────────────┼─────────────┤
  cypress:server:util:process_profiler │    0    │     'Firefox'     │      3       │ '816, 1107, 1161'  │   69.54    │     30.24      │  450.29  │    625.48    │   759.52    │
  cypress:server:util:process_profiler │    1    │     'cypress'     │      1       │        '19'        │    0.31    │     10.42      │  226.39  │    226.62    │    288.3    │
  cypress:server:util:process_profiler │    2    │ 'electron-shared' │      4       │ '27, 184, 28, 219' │     0      │       0        │  208.7   │    189.64    │    208.7    │
  cypress:server:util:process_profiler │    3    │     'plugin'      │      1       │       '252'        │     0      │      0.44      │  129.18  │    134.32    │   142.04    │
  cypress:server:util:process_profiler │    4    │      'other'      │      2       │    '1280, 1281'    │     0      │       0        │   3.4    │     3.36     │     3.4     │
  cypress:server:util:process_profiler │    5    │      'TOTAL'      │      11      │        '-'         │   69.85    │     35.98      │ 1017.96  │   1052.79    │   1326.67   │
  cypress:server:util:process_profiler └─────────┴───────────────────┴──────────────┴────────────────────┴────────────┴────────────────┴──────────┴──────────────┴─────────────┘ +10s
  cypress:server:util:process_profiler current & mean memory and CPU usage by process group:
  cypress:server:util:process_profiler ┌─────────┬───────────────────┬──────────────┬────────────────────┬────────────┬────────────────┬──────────┬──────────────┬─────────────┐
  cypress:server:util:process_profiler │ (index) │       group       │ processCount │        pids        │ cpuPercent │ meanCpuPercent │ memRssMb │ meanMemRssMb │ maxMemRssMb │
  cypress:server:util:process_profiler ├─────────┼───────────────────┼──────────────┼────────────────────┼────────────┼────────────────┼──────────┼──────────────┼─────────────┤
  cypress:server:util:process_profiler │    0    │     'Firefox'     │      3       │ '816, 1107, 1161'  │    0.81    │     25.33      │  448.81  │    596.03    │   759.52    │
  cypress:server:util:process_profiler │    1    │     'cypress'     │      1       │        '19'        │    0.18    │      8.96      │  226.8   │    226.64    │    288.3    │
  cypress:server:util:process_profiler │    2    │ 'electron-shared' │      4       │ '27, 184, 28, 219' │     0      │       0        │  208.7   │    192.36    │    208.7    │
  cypress:server:util:process_profiler │    3    │     'plugin'      │      1       │       '252'        │     0      │      0.36      │  129.18  │    133.47    │   142.04    │
  cypress:server:util:process_profiler │    4    │      'other'      │      2       │    '1309, 1310'    │     0      │       0        │   3.39   │     3.36     │     3.4     │
  cypress:server:util:process_profiler │    5    │      'TOTAL'      │      11      │        '-'         │    0.98    │     30.98      │ 1016.88  │   1047.66    │   1326.67   │
  cypress:server:util:process_profiler └─────────┴───────────────────┴──────────────┴────────────────────┴────────────┴────────────────┴──────────┴──────────────┴─────────────┘ +10s

Then after waiting for like 10 Minutes I got this

cypress:server:server-base Got CONNECT request from firefox.settings.services.mozilla.com:443 +6m
  cypress:https-proxy Writing browserSocket connection headers { url: 'firefox.settings.services.mozilla.com:443', headLength: 0, headers: { 'user-agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:77.0) Gecko/20100101 Firefox/77.0', 'proxy-connection': 'keep-alive', connection: 'keep-alive', host: 'firefox.settings.services.mozilla.com:443' } } +6m
  cypress:https-proxy Got first head bytes { url: 'firefox.settings.services.mozilla.com:443', head: "\x16\x03\x01\x02\x00\x01\x00\x01�\x03\x03�/\x01��9 �ݘ��\x1E, 3bc}�\x11\x12}.u>��K��`�ڍ\x11\x10�" } +6ms
  cypress:https-proxy Making intercepted connection to 36787 +1ms
  cypress:network:connect successfully connected { opts: { port: 36787, host: 'localhost', getDelayMsForRetry: [Function: getDelayForRetry] }, iteration: 0 } +6m
  cypress:https-proxy received upstreamSocket callback for request { port: 36787, hostname: 'localhost', err: undefined } +5ms
  cypress:server:util:socket_allowed allowing socket { localPort: 53150 } +6m
  cypress:proxy:http Entering stage { stage: 'IncomingRequest' } +6m
  cypress:proxy:http Running middleware { stage: 'IncomingRequest', middlewareName: 'LogRequest' } +0ms
  cypress:proxy:http:request-middleware proxying request { req: { method: 'GET', proxiedUrl: 'https://firefox.settings.services.mozilla.com/v1/buckets/monitor/collections/changes/records?', headers: { host: 'firefox.settings.services.mozilla.com', 'user-agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:77.0) Gecko/20100101 Firefox/77.0', accept: '*/*', 'accept-language': 'en-US,en;q=0.5', 'accept-encoding': 'gzip, deflate, br', connection: 'keep-alive' } } } +6m
  cypress:proxy:http Running middleware { stage: 'IncomingRequest', middlewareName: 'MaybeEndRequestWithBufferedResponse' } +1ms
  cypress:proxy:http Running middleware { stage: 'IncomingRequest', middlewareName: 'InterceptRequest' } +0ms
  cypress:net-stubbing:server:intercept-request intercepting request { requestId: 'interceptedRequest230', req: { url: '/v1/buckets/monitor/collections/changes/records?' } } +6m
  cypress:net-stubbing:server:util sending event to driver { eventName: 'before:request', data: { eventId: 'event231', subscription: { eventName: 'before:request', await: true, routeId: '1618506135594-55' }, requestId: 'interceptedRequest230', data: { headers: [Object], url: 'https://firefox.settings.services.mozilla.com/v1/buckets/monitor/collections/changes/records?', method: 'GET', httpVersion: '1.1', body: '' } } } +4ms

After that the current memory and CPU usage is displayed again. Waiting didn't resolve anything. I have to manually stop the container.

I tested for about an hour and I intend to test more, but it seems to be happening with firefox pretty consistently for me. My test with electron was successful (meaning it finished as expected).

Since it will "freeze" on random tests I cannot really give any code to reproduce. Most tests that I saw it freeze didn't do anything crazy. Just some intercepts, visits, shoulds and maybe a type here and there. But I can give some code on how I start cypress in docker.

I basically create a docker image with the following code

FROM cypress/included:7.1.0
USER root
COPY cypress.json /cypress.json
COPY cypress /cypress

Copying the files was done before I was in the project, but mounting the directory via docker run -v didn't have any impact. After creating the image I run the container docker run --name cy-e2e imagename --browser firefox --config video=false

[jelyck]

Same behaviour here. Only seen in firefox.

Not working in 7.1.0 or later (just tested 7.3.0). Not sure about 7.0.1.

Latest working revision is 7.0.0.

Nothing random for me.

Env: Linux Jenkins Docker cypress/included:7.3.0 (using --browser firefox)

Test case:

    it('should login successfully', () => {
      cy.get('[data-cy=account-button]').should('exist').click();
      cy.get('input[name="username"]').clear().type(username);
      cy.get('input[name="password"]').clear().type(password);
  
      cy.intercept({
        method: Cypress.env('endpoints').auth.sessions.method,
        url: Cypress.env('endpoints').auth.sessions.url,
      }).as('session');
  
      cy.intercept({
        method: Cypress.env('endpoints').auth.users.method,
        url: Cypress.env('endpoints').auth.users.url,
      }).as('users');
  
      cy.get('#login').click();
  
      cy.wait(['@session', '@users']).spread((authReqInt, userReqInt) => {
        expect(authReqInt.response.statusCode).to.equal(200);
        expect(authReqInt.response.body).to.not.be.null;
        expect(userReqInt.response.statusCode).to.equal(200);
        expect(userReqInt.response.body).to.not.be.null;
      });
  
      cy.get('[data-cy=welcome-message]').should('exist');
    });

Note: The same code is executed in a previous test which PASS. Second time the code is executed Cypress hangs.

Note2: Works without any issues when running cypress normally outside of docker.

Suspected changes:

• Cypress no longer crashes in certain circumstances when running in Docker without --ipc=host. Fixes cypress-io/cypress#15814 and cypress-io/cypress#350.

• Fixed an issue introduced in 7.0.0 where requests with responses stubbed via cy.intercept(routeMatcher, staticResponse) would still be sent to the destination server. Fixes cypress-io/cypress#15841.

Log file (using cypress/included:7.3.0): docker-firefox-hangs.txt

[jelyck]

Some more information:

Name + Tag	Base image
cypress/included:7.0.0	cypress/browsers:node12.18.3-chrome89-ff86
cypress/included:7.0.1	cypress/browsers:node14.16.0-chrome89-ff77
cypress/included:7.1.0	cypress/browsers:node14.16.0-chrome89-ff77
cypress/included:7.2.0	cypress/browsers:node14.16.0-chrome89-ff77
cypress/included:7.3.0	cypress/browsers:node14.16.0-chrome89-ff77

Seems like the Firefox version used in cypress/included:7.0.1 compared to cypress/included:7.0.0 has been stepped backwards. Maybe an unintentional mistake that was done during the node14 uplift? A cypress/browsers build with Firefox version 86 also exists.

Name + Tag	Base image	Chrome	Firefox	Edge
cypress/browsers:node14.16.0-chrome89-ff77	cypress/base:14.16.0	89.0.4389.72	77.0
cypress/browsers:node14.16.0-chrome89-ff86	cypress/base:14.16.0	89.0.4389.72	86.0.1

Not sure if the older Firefox version could cause the issue with hanging Cypress tests. The issue also seems to be related to shm.

• Cypress no longer crashes in certain circumstances when running in Docker without --ipc=host. Fixes cypress-io/cypress#15814 and cypress-io/cypress#350.

This fix was also added in 7.0.1 and as far as I can see should only affect electron and chrome browsers... However, as seen in the log something called "electron-shared" is running and I'm not sure if --disable-dev-shm-usage could affect firefox.

Also, running docker with increased shm-size=2gb seems to work in Firefox. That is:

$ docker run -it --shm-size=2gb -v $PWD:/e2e -w /e2e cypress/included:7.3.0

With increased shm-size Cypress runs normally without freezing.

[jelyck]

Some more info.

I have tested some different versions.

Versions	Freeze/Working
cypress/browsers:node14.16.0-chrome89-ff86 + Cypress 7.3.0	Working
cypress/browsers:node14.16.0-chrome89-ff86 + Cypress 7.0.0	Working
cypress/browsers:node14.16.0-chrome89-ff86 + Cypress 7.3.0 + --shm-size=2gb	Working
cypress/browsers:node14.16.0-chrome89-ff77 + Cypress 7.3.0	Freeze
cypress/browsers:node14.16.0-chrome89-ff77 + Cypress 7.0.0	Freeze
cypress/browsers:node14.16.0-chrome89-ff77 + Cypress 7.3.0 + --shm-size=2gb	Working
cypress/browsers:node14.16.0-chrome89-ff77 + Cypress 7.0.0 + --shm-size=2gb	Working
cypress/included:7.0.0 (using ff86)	Working
cypress/included:7.0.1 (using ff77)	Freeze
cypress/included:7.3.0 (using ff77)	Freeze
cypress/included:7.3.0 (using ff77) + --shm-size=2gb	Working

So the issue seems to be due to ff77 running in docker with default shm-size.

I suggest updating cypress/included with cypress/browsers:node14.16.0-chrome89-ff86, same Firefox version as in cypress/included:7.0.0, and update the documentation regarding shm-size and why increasing it solves the issue.

https://github.com/cypress-io/cypress/issues/350#issuecomment-814959972 mentions that there is no longer a need to specify --ipc=host or increase /dev/shm size for Chrome and Electron. However, seems like issues still exists with Firefox. Even though ff86 seems OK there might be issues in future versions.

[jennifer-shehane]

I'm going to move this issue to the docker repo since it's specific to the docker images.

[emilyrohrbough]

@jenslycke Can you provide verbose logs with cypress* (without the :)?

[mjhenkes]

Since our current supported browser versions are 86 and above, i'm going to go ahead and close this issue. https://docs.cypress.io/guides/guides/launching-browsers#Browser-versions-supported