"Leaking" EPIPE error breaks retrying

[heypiotr]

Describe the bug

Node.js 16.4. We're seeing this on an AWS Lambda running Amazon Linux 2, but I was also able to repro on WSL2 with Ubuntu 20.04.

The gist of the issue is, an EPIPE error happens that emits an error event directly on the Request object, bypassing any _beforeError and other handling inside a Request, and instead going straight to rejecting the promise.

Our repro path is a bit convoluted, but reliable:

1. We've got http/https agent set up with keepalive.
2. AWS Lambda pauses the process between tasks. When the pause is long enough, I think this can lead to the socket getting torn down, and when the process resumes, an EPIPE error. In my repro, I've just been CTRL+Z'ing the process for 5 minutes, and then resuming it.
3. The got promise gets rejected with an EPIPE, without a retry.
4. (But something is not cleaned up, so in some later tick, got actually attempts a retry, but the promise is already rejected, so it errors with an unrelated "The onCancel handler was attached after the promise settled." error.)

Code to reproduce

https://gist.github.com/heypiotr/f97b3c2069770988a5f4f3785d83f730

What it looks like when I repro it:

• start the script, the first request goes through fine, pause the script

heypiotr at PC in ~/sandbox/got-epipe
$ node index.mjs
requesting 0 0
{
  "args": {},
  "data": "",
  "files": {
    "bar": "bar\n"
  },
  "form": {
    "foo": "foo"
  },
  "headers": {
    "Accept-Encoding": "gzip, deflate, br",
    "Content-Length": "285",
    "Content-Type": "multipart/form-data; boundary=form-data-boundary-41m9vp3q1waroamv",
    "Host": "httpbin.org",
    "User-Agent": "got (https://github.com/sindresorhus/got)",
    "X-Amzn-Trace-Id": "Root=1-621c9052-6c84c7890e0a794b6e14c8d4"
  },
  "json": null,
  "origin": "213.127.39.54",
  "url": "https://httpbin.org/post"
}

sleeping
^Z
[1]+  Stopped                 node index.mjs

• wait 5 minutes
• resume the script, it immediately fails with an EPIPE

heypiotr at PC in ~/sandbox/got-epipe
$ fg
node index.mjs
requesting 0 1
Trace: rejecting Error: write EPIPE
    at Request.onError (file:///home/heypiotr/sandbox/got-epipe/node_modules/got/dist/source/as-promise/index.js:103:13)
    at Object.onceWrapper (node:events:640:26)
    at Request.emit (node:events:520:28)
    at emitErrorNT (node:internal/streams/destroy:157:8)
    at errorOrDestroy (node:internal/streams/destroy:220:7)
    at onwriteError (node:internal/streams/writable:422:3)
    at onwrite (node:internal/streams/writable:457:7)
    at Object.callback (file:///home/heypiotr/sandbox/got-epipe/node_modules/got/dist/source/core/index.js:1025:13)
    at callback (node:internal/streams/writable:552:21)
    at onwriteError (node:internal/streams/writable:415:3)

• ^ this is a console.trace I added before this line, you can see Node.js emits the error event on the Request object directly via errorOrDestroy => emitErrorNT

error Error: write EPIPE
    at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:94:16)
    at writevGeneric (node:internal/stream_base_commons:138:26)
    at TLSSocket.Socket._writeGeneric (node:net:793:11)
    at TLSSocket.Socket._writev (node:net:802:8)
    at doWrite (node:internal/streams/writable:406:12)
    at clearBuffer (node:internal/streams/writable:561:5)
    at TLSSocket.Writable.uncork (node:internal/streams/writable:348:7)
    at ClientRequest._flushOutput (node:_http_outgoing:970:10)
    at ClientRequest._flush (node:_http_outgoing:939:22)
    at onSocketNT (node:_http_client:813:9) {
  errno: -32,
  code: 'EPIPE',
  syscall: 'write'
}

• ^ this is the error caught and printed by the try {} catch {} surrounding the got invocation
• it actually still attempts a retry after a short while, but since the promise already got rejected, it crashes with another error:

sleeping
Retrying request, url=https://httpbin.org/post, retryCount=1, error=RequestError: write EPIPE
file:///home/heypiotr/sandbox/got-epipe/node_modules/p-cancelable/index.js:48
                                        throw new Error('The `onCancel` handler was attached after the promise settled.');
                                              ^

Error: The `onCancel` handler was attached after the promise settled.
    at onCancel (file:///home/heypiotr/sandbox/got-epipe/node_modules/p-cancelable/index.js:48:12)
    at makeRequest (file:///home/heypiotr/sandbox/got-epipe/node_modules/got/dist/source/as-promise/index.js:33:13)
    at Request.<anonymous> (file:///home/heypiotr/sandbox/got-epipe/node_modules/got/dist/source/as-promise/index.js:119:17)
    at Object.onceWrapper (node:events:640:26)
    at Request.emit (node:events:520:28)
    at file:///home/heypiotr/sandbox/got-epipe/node_modules/got/dist/source/core/index.js:388:26

Checklist

• [x] I have read the documentation.
• [x] I have tried my code with the latest version of Node.js and Got.

---

I'd be happy to work on a PR for this, but I could use some pointers on how to attack it.

I think that ideally, the fix would be somewhere inside Request, so that it applies to both the Promise and the Stream APIs.

Could it be as simple as Request listening for its own error events and passing them through _beforeError?

[szmarczak]

Thanks for the report! While I can't reproduce with the example, you've cleanly narrowed the issue down to the point - the error in write callback is emitted directly, regardless of other error handlers. I'll gather more information and provide you with hints on how to fix it ASAP ❤️

[Kit-p]

@szmarczak May I ask if there is any update on this issue? My team has come across the same problem and could not catch the error with try catch.

To provide more context, it only happens on a retry request. My guess would be that the request is cloned before retrying and the on('error', ...) listener is not attached to the cloned request.

Let me know if there is any other information I can provide to move this forward, thank you!