cypress Cypress crashes with "Error: This socket has been ended by the other party" at unpredictable times

Current behavior

Since about July 19th my team has been experiencing a bug with cypress which causes cypress to crash at unpredictable moments when tested on one of our applications. The crash happens as frequently as every 2 minutes of cypress running, or can be as long as >1hr of cypress running.

Version info:

Cypress binary version: 12.9.0 and 13.0.0
Electron version: 21.0.0
Bundled Node version: 16.16.0
Chrome version: 115.0.5790.170

Several members of my team and I have tried to isolate the cause, or even identify specific steps to reproduce but have been unable to get any closer than "run the tests and wait".

We think the timing of this bug may coincide with a chrome version update, but are not sure.

here is a redacted log around the time of the failure, with DEBUG=cypress:* turned on. I can share the un-redacted version of the logs directly with cypress maintainers.

Here is just the error (excerpt from full logs above). The stack trace points to node internals, but it's not clear where it's being called from cypress.

This socket has been ended by the other party
Error: This socket has been ended by the other party
    at TLSSocket.writeAfterFIN [as write] (node:net:487:14)
    at IncomingMessage.ondata (node:internal/streams/readable:754:22)
    at IncomingMessage.emit (node:events:527:28)
    at addChunk (node:internal/streams/readable:315:12)
    at readableAddChunk (node:internal/streams/readable:289:9)
    at Readable.push (node:internal/streams/readable:228:10)
    at HTTPParser.parserOnBody (node:_http_common:141:24)
    at TLSSocket.socketOnData (node:_http_client:494:22)
    at TLSSocket.emit (node:events:527:28)
    at addChunk (node:internal/streams/readable:315:12)
    at readableAddChunk (node:internal/streams/readable:289:9)
    at Readable.push (node:internal/streams/readable:228:10)
    at TLSWrap.onStreamRead (node:internal/stream_base_commons:190:23)

Desired behavior

Cypress should be stable and not crash.

Test code to reproduce

TBD - currently the only way we can reproduce is by running our tests for an average of 5 minutes on a specific app.

Cypress Version

12.9.0

Node version

16.16.0

Operating System

macOS 13.5

Debug Logs

2023-07-26T23:30:57.466Z cypress:server:server-base Got CONNECT request from myapp.redacted.com:443
2023-07-26T23:30:57.466Z cypress:https-proxy Writing browserSocket connection headers { url: 'myapp.redacted.com:443', headLength: 0, headers: { host: 'myapp.redacted.com:443', 'proxy-connection': 'keep-alive', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36' } }
2023-07-26T23:30:57.466Z cypress:server:socket-base backend:request { eventName: 'protocol:command:log:added', args: [ { id: 'log-https://myapp.redacted.com-95', displayName: 'stylesheet', event: true, hookId: 'h7', instrument: 'command', message: 'Object{5}', method: 'GET', name: 'request', renderProps: [Object], state: 'pending', testId: 'r3', timeout: 0, type: 'parent', url: 'https://myapp.redacted.com/assets/redacted/static/packages/web/common/6759373945.css', wallClockStartedAt: '2023-07-26T23:30:57.466Z', testCurrentRetry: 0, timestamp: 1690414257466.5 } ] }
2023-07-26T23:30:57.466Z cypress:protocol:capture:CypressEventsManager inserting command log into database, id: log-https://myapp.redacted.com-95, type: log:added
2023-07-26T23:30:57.467Z cypress:https-proxy Got first head bytes { url: 'myapp.redacted.com:443', head: "\x16\x03\x01\x02:\x01\x00\x026\x03\x03�r��ɪ�@���\x0F\x04؂nJ�_*\\6��z��g��Tl <H+�\\�k�'>0~��\x1D/�\x02uC�]" }
2023-07-26T23:30:57.467Z cypress:https-proxy Making intercepted connection to 61210
2023-07-26T23:30:57.467Z cypress:server:socket-base backend:request { eventName: 'net', args: [ 'event:handler:resolved', { eventId: 'event29284', changedData: [Object], stopPropagation: false } ] }
2023-07-26T23:30:57.467Z cypress:net-stubbing:server:driver-events received driver event { eventName: 'event:handler:resolved', args: [ 'event:handler:resolved', { eventId: 'event29284', changedData: [Object], stopPropagation: false } ] }
2023-07-26T23:30:57.468Z cypress:network:cors Parsed URL { port: '443', protocol: 'https:', subdomain: 'myapp', domain: 'redacted', tld: 'tools' }
2023-07-26T23:30:57.468Z cypress:server:remote-states getting remote state: { auth: null, origin: 'https://myapp.redacted.com', strategy: 'http', fileServer: null, domainName: 'redacted.com', props: { port: '443', protocol: 'https:', subdomain: 'myapp', domain: 'redacted', tld: 'tools' } } for: https://myapp.redacted.com/assets/airbnb/static/packages/web/common/6759373945.css
2023-07-26T23:30:57.468Z cypress:network:cors Parsed URL { port: '443', protocol: 'https:', subdomain: null, domain: 'redacted', tld: 'tools' }
2023-07-26T23:30:57.468Z cypress:server:remote-states getting remote state: { auth: null, origin: 'https://myapp.redacted.com', strategy: 'http', fileServer: null, domainName: 'redacted.com', props: { port: '443', protocol: 'https:', subdomain: 'myapp', domain: 'redacted', tld: 'tools' } } for: https://redacted.com
2023-07-26T23:30:57.468Z cypress:server:stream_buffer stream buffer writeable final called
2023-07-26T23:30:57.468Z cypress:network:agent addRequest called { isHttps: true, href: 'https://myapp.redacted.com/assets/redacted/static/packages/web/common/6759373945.css' }
2023-07-26T23:30:57.468Z cypress:network:agent got family { family: 6, href: 'https://myapp.redacted.com/assets/redacted/static/packages/web/common/6759373945.css' }
2023-07-26T23:30:57.469Z cypress:network:connect successfully connected { opts: { port: 61210, host: 'localhost', getDelayMsForRetry: [Function: h] }, iteration: 0 }
2023-07-26T23:30:57.469Z cypress:https-proxy received upstreamSocket callback for request { port: 61210, hostname: 'localhost', err: undefined }
2023-07-26T23:30:57.469Z cypress:server:util:socket_allowed allowing socket { localPort: 65365 }
2023-07-26T23:30:57.470Z cypress:https-proxy upgrade /pikachu/ws2/123123123123?sendSeq=true
2023-07-26T23:30:57.470Z cypress:server:server-base Got UPGRADE request from /pikachu/ws2/123123123123?sendSeq=true
2023-07-26T23:30:57.470Z cypress:network:cors Parsed URL { port: '443', protocol: 'https:', subdomain: 'myapp', domain: 'redacted', tld: 'tools' }
2023-07-26T23:30:57.470Z cypress:network:agent addRequest called { isHttps: true, href: 'https://myapp.redacted.com:443/pikachu/ws2/123123123123?sendSeq=true' }
2023-07-26T23:30:57.470Z cypress:network:agent got family { family: 6, href: 'https://myapp.redacted.com:443/pikachu/ws2/123123123123?sendSeq=true' }
2023-07-26T23:30:57.472Z cypress:server:socket-base backend:request { eventName: 'protocol:command:log:changed', args: [ { id: 'log-https://myapp.redacted.com-6', hookId: 'h2', instrument: 'route', isStubbed: true, method: '*', name: 'route', numResponses: 117, renderProps: {}, state: 'pending', testId: 'r3', url: '{hostname: /\\.(redacted\\.com|redacted\\.com|staging\\.redacted\\.com)$/}', wallClockStartedAt: '2023-07-26T23:30:46.783Z', testCurrentRetry: 0, snapshots: [Array], timestamp: 1690414257467.4 } ] }
2023-07-26T23:30:57.472Z cypress:protocol:capture:CypressEventsManager inserting command log into database, id: log-https://myapp.redacted.com-6, type: log:changed
2023-07-26T23:30:57.473Z cypress:server:socket-base backend:request { eventName: 'protocol:command:log:changed', args: [ { id: 'log-https://myapp.redacted.com-95', displayName: 'stylesheet', event: true, hookId: 'h7', instrument: 'command', message: 'Object{5}', method: 'GET', name: 'request', renderProps: [Object], state: 'pending', testId: 'r3', timeout: 0, type: 'parent', url: 'https://myapp.redacted.com/assets/redacted/static/packages/web/common/6759373945.css', wallClockStartedAt: '2023-07-26T23:30:57.466Z', testCurrentRetry: 0, snapshots: [Array], timestamp: 1690414257467.5 } ] }
2023-07-26T23:30:57.473Z cypress:protocol:capture:CypressEventsManager inserting command log into database, id: log-https://myapp.redacted.com-95, type: log:changed
2023-07-26T23:30:57.493Z cypress:net-stubbing:server:util sending event to driver { eventName: 'before:request', data: { eventId: 'event29287', subscription: { eventName: 'before:request', await: true, routeId: '1690414246783-4' }, browserRequestId: undefined, requestId: 'interceptedRequest29286', data: { headers: [Object], url: 'https://myapp.redacted.com/proxy/pikachu/api/v2/websocket_tokens?locale=en', method: 'POST', httpVersion: '1.1', resourceType: undefined, query: [Object], body: '{}' } } }
2023-07-26T23:30:57.496Z cypress:server:socket-base backend:request { eventName: 'net', args: [ 'event:handler:resolved', { eventId: 'event29287', changedData: [Object], stopPropagation: false } ] }
2023-07-26T23:30:57.496Z cypress:net-stubbing:server:driver-events received driver event { eventName: 'event:handler:resolved', args: [ 'event:handler:resolved', { eventId: 'event29287', changedData: [Object], stopPropagation: false } ] }
2023-07-26T23:30:57.496Z cypress:network:cors Parsed URL { port: '443', protocol: 'https:', subdomain: 'myapp', domain: 'redacted', tld: 'tools' }
2023-07-26T23:30:57.496Z cypress:server:remote-states getting remote state: { auth: null, origin: 'https://myapp.redacted.com', strategy: 'http', fileServer: null, domainName: 'redacted.com', props: { port: '443', protocol: 'https:', subdomain: 'myapp', domain: 'redacted', tld: 'tools' } } for: https://myapp.redacted.com/proxy/pikachu/api/v2/websocket_tokens?locale=en
2023-07-26T23:30:57.496Z cypress:network:cors Parsed URL { port: '443', protocol: 'https:', subdomain: null, domain: 'redacted', tld: 'tools' }
2023-07-26T23:30:57.496Z cypress:server:remote-states getting remote state: { auth: null, origin: 'https://myapp.redacted.com', strategy: 'http', fileServer: null, domainName: 'redacted.com', props: { port: '443', protocol: 'https:', subdomain: 'myapp', domain: 'redacted', tld: 'tools' } } for: https://redacted.com
2023-07-26T23:30:57.497Z cypress:network:agent addRequest called { isHttps: true, href: 'https://myapp.redacted.com/proxy/pikachu/api/v2/websocket_tokens?locale=en' }
2023-07-26T23:30:57.497Z cypress:network:agent got family { family: 6, href: 'https://myapp.redacted.com/proxy/pikachu/api/v2/websocket_tokens?locale=en' }
2023-07-26T23:30:57.521Z cypress:server:socket-base backend:request { eventName: 'protocol:command:log:changed', args: [ { id: 'log-https://myapp.redacted.com-6', hookId: 'h2', instrument: 'route', isStubbed: true, method: '*', name: 'route', numResponses: 118, renderProps: {}, state: 'pending', testId: 'r3', url: '{hostname: /\\.(redacted\\.com|redacted\\.com|staging\\.redacted\\.com)$/}', wallClockStartedAt: '2023-07-26T23:30:46.783Z', testCurrentRetry: 0, snapshots: [Array], timestamp: 1690414257495.8 } ] }
2023-07-26T23:30:57.521Z cypress:protocol:capture:CypressEventsManager inserting command log into database, id: log-https://myapp.redacted.com-6, type: log:changed
2023-07-26T23:30:57.583Z cypress:server:request received status code & headers on request { requestId: 'request29275', statusCode: 200, headers: { 'content-type': 'application/json' } }
2023-07-26T23:30:57.583Z cypress:server:request successful response received { requestId: 'request29275' }
2023-07-26T23:30:57.584Z cypress:network:cors Parsed URL { port: '443', protocol: 'https:', subdomain: null, domain: 'redacted', tld: 'tools' }
2023-07-26T23:30:57.584Z cypress:server:remote-states getting remote state: { auth: null, origin: 'https://myapp.redacted.com', strategy: 'http', fileServer: null, domainName: 'redacted.com', props: { port: '443', protocol: 'https:', subdomain: 'myapp', domain: 'redacted', tld: 'tools' } } for: https://redacted.com
2023-07-26T23:30:57.584Z cypress:network:cors Parsed URL { port: '443', protocol: 'https:', subdomain: 'rum-http-intake.logs', domain: 'metricsservice', tld: 'com' }
2023-07-26T23:30:57.584Z cypress:network:cors Parsed URL { port: '443', protocol: 'https:', subdomain: 'rum-http-intake.logs', domain: 'metricsservice', tld: 'com' }
2023-07-26T23:30:57.584Z cypress:server:remote-states getting primary remote state: { auth: null, origin: 'https://myapp.redacted.com', strategy: 'http', fileServer: null, domainName: 'redacted.com', props: { port: '443', protocol: 'https:', subdomain: 'myapp', domain: 'redacted', tld: 'tools' } }
2023-07-26T23:30:57.584Z cypress:network:cors Parsed URL { port: '443', protocol: 'https:', subdomain: 'rum-http-intake.logs', domain: 'metricsservice', tld: 'com' }
2023-07-26T23:30:57.584Z cypress:network:cors Parsed URL { port: '443', protocol: 'https:', subdomain: 'rum-http-intake.logs', domain: 'metricsservice', tld: 'com' }
2023-07-26T23:30:57.584Z cypress:network:cors Parsed URL { port: '443', protocol: 'https:', subdomain: null, domain: 'redacted', tld: 'tools' }
2023-07-26T23:30:57.584Z cypress:server:remote-states getting remote state: { auth: null, origin: 'https://myapp.redacted.com', strategy: 'http', fileServer: null, domainName: 'redacted.com', props: { port: '443', protocol: 'https:', subdomain: 'myapp', domain: 'redacted', tld: 'tools' } } for: https://redacted.com
2023-07-26T23:30:57.585Z cypress:protocol:capture:Tt Insertion attempt for network:response-received-extra-info
2023-07-26T23:30:57.585Z cypress:protocol:capture:Tt Inserting network:response-received-extra-info into events table with runnableId undefined, attempt undefined
2023-07-26T23:30:57.586Z cypress:protocol:capture:Tt Insertion attempt for network:response-received
2023-07-26T23:30:57.586Z cypress:protocol:capture:Tt Inserting network:response-received into events table with runnableId undefined, attempt undefined
2023-07-26T23:30:57.587Z cypress:protocol:capture:Tt Insertion attempt for network:loading-finished
2023-07-26T23:30:57.587Z cypress:protocol:capture:Tt Inserting network:loading-finished into events table with runnableId undefined, attempt undefined
2023-07-26T23:30:57.587Z cypress:protocol:capture:NetworkManager Network.loadingFinished {
  request: {
    requestId: '71708.460',
    requestURLs: [
      'https://redacted2.com'
    ],
    type: 'Ping',
    requestWillBeSentTimestamp: 153491.587811,
    requestWillBeSentWallTime: 1690414256581.82
  },
  response: {
    requestId: '71708.460',
    mimeType: 'application/json',
    type: 'Ping'
  }
}
2023-07-26T23:30:57.596Z cypress:server:request received status code & headers on request { requestId: 'request29266', statusCode: 200, headers: { 'content-type': 'application/json;charset=utf-8' } }
2023-07-26T23:30:57.596Z cypress:server:request successful response received { requestId: 'request29266' }
2023-07-26T23:30:57.597Z cypress:net-stubbing:server:intercept-response InterceptResponse { req: { url: 'https://myapp.redacted.com/proxy/service1/api/v1/providers?locale=en' }, request: a { subscriptionsByRoute: [ [Object] ], includeBodyInAfterResponse: false, responseSent: false, onResponse: [Function (anonymous)], id: 'interceptedRequest29264', req: IncomingMessage { _readableState: [ReadableState], _events: [Object: null prototype], _eventsCount: 1, _maxListeners: undefined, socket: [TLSSocket], httpVersionMajor: 1, httpVersionMinor: 1, httpVersion: '1.1', complete: true, rawHeaders: [Array], rawTrailers: [], aborted: false, upgrade: false, url: 'https://myapp.redacted.com/proxy/service1/api/v1/providers?locale=en', method: 'GET', statusCode: null, statusMessage: null, client: [TLSSocket], _consuming: false, _dumped: false, proxiedUrl: 'https://myapp.redacted.com/proxy/service1/api/v1/providers?locale=en', next: [Function: h], baseUrl: '', originalUrl: '/proxy/service1/api/v1/providers?locale=en', _parsedUrl: [Url], params: [Object], query: [Object], res: [ServerResponse], secret: undefined, cookies: [Object], signedCookies: [Object: null prototype] {}, route: [p], isAUTFrame: false, browserPreRequest: undefined, resourceType: undefined, matchingRoutes: [Array], requestId: 'interceptedRequest29264', body: '', responseTimeout: 60000, [Symbol(kCapture)]: false, [Symbol(kHeaders)]: [Object], [Symbol(kHeadersCount)]: 30, [Symbol(kTrailers)]: null, [Symbol(kTrailersCount)]: 0, [Symbol(RequestTimeout)]: undefined }, res: ServerResponse { _events: [Object: null prototype], _eventsCount: 2, _maxListeners: undefined, outputData: [], outputSize: 0, writable: true, destroyed: false, _last: false, chunkedEncoding: false, shouldKeepAlive: true, maxRequestsOnConnectionReached: false, _defaultKeepAlive: true, useChunkedEncodingByDefault: true, sendDate: true, _removedConnection: false, _removedContLen: false, _removedTE: false, _contentLength: null, _hasBody: true, _trailer: '', finished: false, _headerSent: false, _closed: false, socket: [TLSSocket], _header: null, _keepAliveTimeout: 5000, _onPendingData: [Function: bound updateOutgoingData], req: [IncomingMessage], _sent100: false, _expect_continue: false, locals: [Object: null prototype] {}, flush: [Function (anonymous)], write: [Function (anonymous)], end: [Function (anonymous)], on: [Function (anonymous)], writeHead: [Function (anonymous)], [Symbol(kCapture)]: false, [Symbol(kNeedDrain)]: false, [Symbol(corked)]: 0, [Symbol(kOutHeaders)]: null }, continueRequest: [Function: next], onError: [Function: _], _onResponse: [Function: onResponse], state: { requests: [Object], routes: [Array], pendingEventHandlers: {}, reset: [Function: reset] }, socket: p { ensureProp: [Function: d], inRunMode: true, supportsRunEvents: true, ended: true, localBus: [EventEmitter], onTestFileChange: [Function: bound ], testFilePath: 'integration/attachments/referenceAttachment.test.ts', onStudioTestFileChange: [Function: bound onStudioTestFileChange], removeOnStudioTestFileChange: [Function: bound removeOnStudioTestFileChange], _io: [u], _protocolManager: [S], _sendResetBrowserTabsForNextTestMessage: [AsyncFunction (anonymous)], _sendResetBrowserStateMessage: [AsyncFunction (anonymous)], _sendFocusBrowserMessage: [AsyncFunction (anonymous)], _isRunnerSocketConnected: [Function (anonymous)] }, lastEvent: 'before:request' } }
2023-07-26T23:30:57.597Z cypress:net-stubbing:server:util sending event to driver { eventName: 'response:callback', data: { eventId: 'event29289', subscription: { eventName: 'response:callback', await: false, routeId: '1690414246783-4' }, browserRequestId: undefined, requestId: 'interceptedRequest29264', data: { headers: [Object], url: 'https://myapp.redacted.com/proxy/service1/api/v1/providers?locale=en', method: null, httpVersion: '1.1', statusCode: 200, statusMessage: 'OK', body: '{"providers":[{"id":"genesys.prod.us-onprem","name":"(OnPrem) Genesys US","disabled":false},{"id":"genesys.prod.eu-onprem","name":"(OnPrem) Genesys EU","disabled":false}],"metadata":{}}' } } }
2023-07-26T23:30:57.597Z cypress:network:cors Parsed URL { port: '443', protocol: 'https:', subdomain: null, domain: 'redacted', tld: 'tools' }
2023-07-26T23:30:57.597Z cypress:server:remote-states getting remote state: { auth: null, origin: 'https://myapp.redacted.com', strategy: 'http', fileServer: null, domainName: 'redacted.com', props: { port: '443', protocol: 'https:', subdomain: 'myapp', domain: 'redacted', tld: 'tools' } } for: https://redacted.com
2023-07-26T23:30:57.597Z cypress:network:cors Parsed URL { port: '443', protocol: 'https:', subdomain: 'myapp', domain: 'redacted', tld: 'tools' }
2023-07-26T23:30:57.597Z cypress:network:cors Parsed URL { port: '443', protocol: 'https:', subdomain: 'myapp', domain: 'redacted', tld: 'tools' }
2023-07-26T23:30:57.597Z cypress:server:remote-states getting primary remote state: { auth: null, origin: 'https://myapp.redacted.com', strategy: 'http', fileServer: null, domainName: 'redacted.com', props: { port: '443', protocol: 'https:', subdomain: 'myapp', domain: 'redacted', tld: 'tools' } }
2023-07-26T23:30:57.597Z cypress:network:cors Parsed URL { port: '443', protocol: 'https:', subdomain: 'myapp', domain: 'redacted', tld: 'tools' }
2023-07-26T23:30:57.597Z cypress:network:cors Parsed URL { port: '443', protocol: 'https:', subdomain: 'myapp', domain: 'redacted', tld: 'tools' }
2023-07-26T23:30:57.598Z cypress:network:cors Parsed URL { port: '443', protocol: 'https:', subdomain: null, domain: 'redacted', tld: 'tools' }
2023-07-26T23:30:57.598Z cypress:server:remote-states getting remote state: { auth: null, origin: 'https://myapp.redacted.com', strategy: 'http', fileServer: null, domainName: 'redacted.com', props: { port: '443', protocol: 'https:', subdomain: 'myapp', domain: 'redacted', tld: 'tools' } } for: https://redacted.com
2023-07-26T23:30:57.598Z cypress:net-stubbing:server:util sending event to driver { eventName: 'after:response', data: { eventId: 'event29290', subscription: { eventName: 'after:response', await: false, routeId: '1690414246783-4' }, browserRequestId: undefined, requestId: 'interceptedRequest29264', data: {} } }
This socket has been ended by the other party
Error: This socket has been ended by the other party
    at TLSSocket.writeAfterFIN [as write] (node:net:487:14)
    at IncomingMessage.ondata (node:internal/streams/readable:754:22)
    at IncomingMessage.emit (node:events:527:28)
    at addChunk (node:internal/streams/readable:315:12)
    at readableAddChunk (node:internal/streams/readable:289:9)
    at Readable.push (node:internal/streams/readable:228:10)
    at HTTPParser.parserOnBody (node:_http_common:141:24)
    at TLSSocket.socketOnData (node:_http_client:494:22)
    at TLSSocket.emit (node:events:527:28)
    at addChunk (node:internal/streams/readable:315:12)
    at readableAddChunk (node:internal/streams/readable:289:9)
    at Readable.push (node:internal/streams/readable:228:10)
    at TLSWrap.onStreamRead (node:internal/stream_base_commons:190:23)
2023-07-26T23:30:57.612Z cypress:server:util:socket_allowed allowed socket closed, removing { localPort: 65365 }
2023-07-26T23:30:57.615Z cypress:server:cloud:api request to url: POST https://api.cypress.io/exceptions with params: {"body":{"err":{"name":"Error","message":"This socket has been ended by the other party","stack":"Error: This socket has been ended by the other party\n    at TLSSocket.writeAfterFIN [as write] (node:net:487:14)\n    at IncomingMessage.ondata (node:internal<stripped-path>readable:754:22)\n    at IncomingMessage.emit (node:events:527:28)\n    at addChunk (node:internal<stripped-path>readable:315:12)\n    at readableAddChunk (node:internal<stripped-path>readable:289:9)\n    at Readable.push (node:internal<stripped-path>readable:228:10)\n    at HTTPParser.parserOnBody (node:_http_common:141:24)\n    at TLSSocket.socketOnData (node:_http_client:494:22)\n    at TLSSocket.emit (node:events:527:28)\n    at addChunk (node:internal<stripped-path>readable:315:12)\n    at readableAddChunk (node:internal<stripped-path>readable:289:9)\n    at Readable.push (node:internal<stripped-path>readable:228:10)\n    at TLSWrap.onStreamRead (node:internal<stripped-path>stream_base_commons:190:23)"},"version":"13.0.0","osName":"darwin","osVersion":"22.5.0","osCpus":[{"model":"Apple M1 Max","speed":24,"times":{"user":36021970,"nice":0,"sys":26155750,"idle":90590640,"irq":0}},{"model":"Apple M1 Max","speed":24,"times":{"user":34591350,"nice":0,"sys":24824410,"idle":93378870,"irq":0}},{"model":"Apple M1 Max","speed":24,"times":{"user":23700240,"nice":0,"sys":7837220,"idle":121410670,"irq":0}},{"model":"Apple M1 Max","speed":24,"times":{"user":12778660,"nice":0,"sys":4449630,"idle":135883380,"irq":0}},{"model":"Apple M1 Max","speed":24,"times":{"user":8386000,"nice":0,"sys":2587290,"idle":142248740,"irq":0}},{"model":"Apple M1 Max","speed":24,"times":{"user":5893610,"nice":0,"sys":1804560,"idle":145575610,"irq":0}},{"model":"Apple M1 Max","speed":24,"times":{"user":7076360,"nice":0,"sys":1555490,"idle":144721150,"irq":0}},{"model":"Apple M1 Max","speed":24,"times":{"user":3503230,"nice":0,"sys":777180,"idle":149130960,"irq":0}},{"model":"Apple M1 Max","speed":24,"times":{"user":2258230,"nice":0,"sys":507080,"idle":150675170,"irq":0}},{"model":"Apple M1 Max","speed":24,"times":{"user":1862440,"nice":0,"sys":420160,"idle":151166990,"irq":0}}],"osMemory":{"free":1599242240,"total":34359738368}},"headers":{"x-os-name":"darwin","x-cypress-version":"13.0.0"}} and token: undefined
2023-07-26T23:30:57.615Z cypress:network:agent addRequest called { isHttps: true, href: 'https://api.cypress.io/exceptions' }
2023-07-26T23:30:57.616Z cypress:network:agent got family { family: 4, href: 'https://api.cypress.io/exceptions' }
2023-07-26T23:30:57.616Z cypress:server:browsers killing browser process
2023-07-26T23:30:57.616Z cypress:server:browsers:chrome closing remote interface client
2023-07-26T23:30:57.616Z cypress:server:browsers:chrome closing chrome
2023-07-26T23:30:57.617Z cypress:proxy:http:util:prerequests metrics: { browserPreRequestsReceived: 7848, proxyRequestsReceived: 4529, immediatelyMatchedRequests: 3312, unmatchedRequests: 501, unmatchedPreRequests: 3743 }
  cypress:cli child event fired { event: 'exit', code: 1, signal: null } +9m
  cypress:cli child event fired { event: 'close', code: 1, signal: null } +1ms

Other

I am hoping that someone from the cypress development community can take a look at the logs here and help us isolate the failure further, or propose potential root causes.

Aug 14 '23 22:08 nickgrout

Confirming this behavior (I'm on the same team @nickgrout).

A few additional things I can add in the hopes they are useful:

error + stacktrace I'm seeing is identical (when it happens -- as nick mentioned, it's sporadic)
interestingly, a similar error + stacktrace does pop up at other times in the debug output, and cypress seems to handle it gracefully. for example:

cypress:https-proxy received error on client browserSocket { err: Error: This socket has been ended by the other party     at Socket.writeAfterFIN [as write] (node:net:487:14)     at Socket.ondata (node:internal/streams/readable:754:22)     at Socket.emit (node:events:527:28)     at addChunk (node:internal/streams/readable:315:12)     at readableAddChunk (node:internal/streams/readable:289:9)     at Readable.push (node:internal/streams/readable:228:10)     at TCP.onStreamRead (node:internal/stream_base_commons:190:23) { code: 'EPIPE' }, url: '[REDACTED]' } +240ms

and

cypress:https-proxy received error on client browserSocket { err: Error: This socket has been ended by the other party     at Socket.writeAfterFIN [as write] (node:net:487:14)     at Socket.ondata (node:internal/streams/readable:754:22)     at Socket.emit (node:events:527:28)     at addChunk (node:internal/streams/readable:315:12)     at readableAddChunk (node:internal/streams/readable:289:9)     at Readable.push (node:internal/streams/readable:228:10)     at TCP.onStreamRead (node:internal/stream_base_commons:190:23) { code: 'EPIPE' }, url: '[REDACTED]' } +6s

these lines are from the a cypress run that eventually crashed, but appear long before the fatal error pops up

the stacktrace is similar to the issue described here https://github.com/cypress-io/cypress/issues/17288, with one important difference: nothing in our stacktrace links back to anything in cypress, whereas the trace in that links back to http-proxy/lib/http-proxy/passes/ws-incoming.js (which is where the problem and workaround for that issue is.).
just want to explicitly call this out: in (2) we're seeing some related logging in cypress:https-proxy, and (3) is a very similar looking trace related to http-proxy. so a couple datapoints that seem to point towards the http(s) proxies being involved somehow..?

from my perspective, the biggest immediate problem is that we don't have more information about the error: as far as we're able to decipher (maybe you'll have more luck with the debug logs!) it's not tracing back to anything in our test code or in cypress; it's just printing an internal node stacktrace and then dying. there very well may be something happening on our end that is triggering the crash -- but we currently don't have enough information to figure out what that is.

at minimum, cypress should probably fail more gracefully / with more info in this scenario (whatever the scenario turns out to be) and ideally, it'd be able to recover.

Aug 15 '23 00:08 joshkaplan

Hi @nickgrout and @joshkaplan , this is related to a POC you all are doing at the moment correct? I have passed this along to our support team who will be reaching out to help you narrow down this issue into something that is a bit easier for us to investigate. At the moment the only thing that is standing out for me is I see two different domains in the logs https://myapp.redacted.com and https://redacted2.com. Without being able to see the tests that are running on which app it's hard to narrow it down much further. Are you visiting multiple domains in this test?

Aug 15 '23 13:08 nagash77

I have passed this along to our support team who will be reaching out to help you narrow down this issue

sounds great thanks!

this is related to a POC you all are doing at the moment correct

not directly related -- these are crashes that started happening on a project that has been in use for a ~year or more. (but yes we are currently running a POC evaluating Cypress Cloud -- and these crashes do make cypress unusable for this project)

At the moment the only thing that is standing out for me is I see two different domains in the logs https://myapp.redacted.com and https://redacted2.com. Without being able to see the tests that are running on which app it's hard to narrow it down much further. Are you visiting multiple domains in this test?

Yes, i believe these tests are visiting multiple domains. The domains and apps are not public, so we have redacted some info. As @nickgrout mentioned above, we can share un-redacted logs in a non-public channel if that's helpful

Aug 15 '23 17:08 joshkaplan

One quick additional note: we may have narrowed down the crash to coming from cy.intercept(). Is this something you my have seen before?

We're currently confirming if changing our use of intercept solves the issue. We'll follow up with more details here once confirmed.

Aug 15 '23 17:08 joshkaplan

Updates

Adjusting/narrowing our use of cy.intercept does seem to resolve.
We did not notice that some very large requests (fetching JS bundles) were being intercepted, and that seemed to be correlated with the crash. But definitely not 100% certain on that.

So, it's mitigated for now -- but would still like to better understand going forward exactly what triggered the crash, and why

Aug 16 '23 16:08 joshkaplan

We are also experiencing this issue mainly in our Devops Pipelines, but not all the time. It also seems to occur way more often during the runs in the pipeline, compared to running the tests locally. We are running Cypress version 12.17.3

@joshkaplan could you explain or show what you mean by adjusting/narrowing the cy.intercept to mitigate the issue?

Sep 07 '23 11:09 lambertsj

@lambertsj sure!

we had a very broad cy.intercept, something like this:

cy.intercept(VERY_BROAD_REGEX, (req) => {
  // a callback to modify the req headers for auth purposes
});

this was matching a lot of requests, including very large assets (e.g. JS, CSS) requests. we removed this intercept, and did auth using a different mechanism. that mitigated the issue, although we still don't fully understand exactly why / what the specific problem was.

Sep 19 '23 18:09 joshkaplan

Do we have any findings on this, as this issue is still there where cypress tests fail with this error- This socket has been ended by the other party Error: This socket has been ended by the other party at TLSSocket.writeAfterFIN [as write] (node:net:541:14) at IncomingMessage.ondata (node:internal/streams/readable:766:22) at IncomingMessage.emit (node:events:513:28) at addChunk (node:internal/streams/readable:324:12) at readableAddChunk (node:internal/streams/readable:297:9) at Readable.push (node:internal/streams/readable:234:10) at HTTPParser.parserOnBody (node:_http_common:131:24) at TLSSocket.socketOnData (node:_http_client:542:22) at TLSSocket.emit (node:events:513:28) at addChunk (node:internal/streams/readable:324:12) at readableAddChunk (node:internal/streams/readable:297:9) at Readable.push (node:internal/streams/readable:234:10) at TLSWrap.onStreamRead (node:internal/stream_base_commons:190:23)

Jan 11 '24 17:01 akshaygupta5233

@akshaygupta5233 I've had success with https://github.com/cypress-io/node-http-proxy/pull/1#issuecomment-1888056120

Jan 11 '24 22:01 nanek

HI @nanek . Please help me with how you did that, I am unable to perform the same and follow the steps!

Jan 12 '24 07:01 akshaygupta5233

@akshaygupta5233 I copied the entire file, ws-incoming.js, from the PR into my repo at cypress/patches/ws-incoming.js.

Cypress will install itself somewhere depending on the operating system and cypress config. In my case I am using github actions, and I've set CYPRESS_CACHE_FOLDER: ~/cypress/cache. So to apply this fix, I run cp cypress/patches/ws-incoming.js ~/cypress/cache/13.6.1/Cypress/resources/app/node_modules/http-proxy-middleware/node_modules/http-proxy/lib/http-proxy/passes/ws-incoming.js before my tests run.

You will need to adjust these paths as needed for your system. Good luck!

Jan 12 '24 23:01 nanek

Hi @nanek I followed the same steps like you said. But still experiencing the same issue. I have hundreds of Test cases to run, and each flow is some 8 to 9 minutes long. I track the RAM usage too, it slowly increases to 15 GB something and then it crashes with the below error I have the MacBook Pro 2022 16 GB. How to run in Cypress with these many tests, or Cypress is not capable of handling these loads? Error: This socket has been ended by the other party Error: This socket has been ended by the other party at TLSSocket.writeAfterFIN [as write] (node:net:541:14) at IncomingMessage.ondata (node:internal/streams/readable:766:22) at IncomingMessage.emit (node:events:513:28) at addChunk (node:internal/streams/readable:324:12) at readableAddChunk (node:internal/streams/readable:297:9) at Readable.push (node:internal/streams/readable:234:10) at HTTPParser.parserOnBody (node:_http_common:131:24) at TLSSocket.socketOnData (node:_http_client:542:22) at TLSSocket.emit (node:events:513:28) at addChunk (node:internal/streams/readable:324:12) at readableAddChunk (node:internal/streams/readable:297:9) at Readable.push (node:internal/streams/readable:234:10) at TLSWrap.onStreamRead (node:internal/stream_base_commons:190:23) [87176:0113/115746.565905:ERROR:ssl_client_socket_impl.cc(978)] handshake failed; returned -1, SSL error code 1, net_error -100 [87176:0113/115746.566789:ERROR:ssl_client_socket_impl.cc(978)] handshake failed; returned -1, SSL error code 1, net_error -100 [87176:0113/115746.567178:ERROR:ssl_client_socket_impl.cc(978)] handshake failed; returned -1, SSL error code 1, net_error -100 [87176:0113/115746.567422:ERROR:ssl_client_socket_impl.cc(978)] handshake failed; returned -1, SSL error code 1, net_error -100 [87176:0113/115746.567500:ERROR:ssl_client_socket_impl.cc(978)] handshake failed; returned -1, SSL error code 1, net_error -101

Jan 13 '24 06:01 akshaygupta5233

This is still an issue so I went ahead and made a PR to the main Cypress repository to validate the proposed fix doesn't break existing proxy workflows here: https://github.com/cypress-io/cypress/pull/29453

May 01 '24 17:05 JamesNimlos

Released in 13.10.0.

This comment thread has been locked. If you are still experiencing this issue after upgrading to Cypress v13.10.0, please open a new issue.

May 21 '24 20:05 cypress-bot[bot]

cypress cypress copied to clipboard

Cypress crashes with "Error: This socket has been ended by the other party" at unpredictable times

Current behavior

Desired behavior

Test code to reproduce

Cypress Version

Node version

Operating System

Debug Logs

Other

cypress
cypress copied to clipboard