Uncatchable `IO error: Broken pipe (os error 32)` when sending to much data through WebSocket

[soootaleb]

Hi guys !

Tested in Deno 1.20.5

So there are a lot of broken pipe problems around and a lot of them are workable via try or Promise.catch (from other posts I found);

But after further investigation in my own app, I drilled down an error that I couldn't catch in any way. I identified that it's happening when there is too much data sent in a WebSocket. Since it's not catchable I think it's a bug, but if you expect me to implement the verification then... (but that would be weird regarding the arbitrary limit that has nothing to do with Deno userland => see reprod)

Here is the reprod (check comment to toggle the bug)

// Typical web server
const server = await Deno.listen({ port: 8080 });

// Delaying the client connection to make sure server is up & running
setTimeout(() => {
  const sock = new WebSocket("ws://localhost:8080")

  sock.onopen = () => {
    console.log("clientop")
  }

  sock.onmessage = () => {
    console.log("recv");
    Deno.exit(0)
  }
}, 1000);

// Wait for a request
for await (const conn of server) {
  Deno.serveHttp(conn)
    .nextRequest()
    .then((event: Deno.RequestEvent | null) => {
      if (event) {
        const { socket, response } = Deno.upgradeWebSocket(event.request)
        
        // string of length Math.pow(2, 24) + 1 WILL WORK but with Math.pow(2, 24) + 2 we trigger the broken pipe
        // I have to admit I don't know why this limit exactly but... found it
        const content = new Array(Math.pow(2, 24) + 2).join(":")

        socket.onopen = () => {
          console.log("open");

          // Wait a bit to fullsend...
          setTimeout(() => {
            console.log("send");

            // try won't do anything, and wrapping in Promise to .catch won't either
            try {
              socket.send(content);
            } catch (error) {
              console.error("Catch")
            }
            
          }, 100);
        }
        
        event.respondWith(response);
      }
    })
}

I didn't reproduce the bug using only the HTTP without upgrading

Tell me if you need more info, cheers

[soootaleb]

I've been conducted some tests for hours and I have to admit it's very hard to give a precise reproduction situation (the one I provide is still legit though, but verifying the data size does not fix everything in my app)

I've run deno with debug logs, and I see that I get (in that order)

DEBUG RS - tungstenite::protocol:555 - Received close frame: Some(CloseFrame { code: Away, reason: "" })
DEBUG RS - tungstenite::protocol:572 - Replying to close with Frame { header: FrameHeader { is_final: true, rsv1: false, rsv2: false, rsv3: false, opcode: Control(Close), mask: None }, payload: [3, 233] }

and then

Error 'IO error: Broken pipe (os error 32)' contains boxed error of unknown type:
  Io(Os { code: 32, kind: BrokenPipe, message: "Broken pipe" })
  Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }

[irbull]

I have reproduced this same error, but not with the size of the data, but by reaching the idleTimeout and having the server close the connection. I've opened https://github.com/denoland/deno/issues/15091 to track

[soootaleb]

@irbull Thanks

I have to admit those BrokenPipe problems that seem uncatchable are really anoying.

Unfortunately no one answered here... hope you're up gives it another chance

[Muha0644]

I am also experiencing this error. I have a webserver that streams videos, and its practically unusable. I experience this crash whenever I try to skip a part of the video (or even when loading it sometimes). Here is what Deno says when it crashes:

error: Uncaught (in promise) BrokenPipe: Broken pipe (os error 32)
          numBytesWritten = await this.writer.write(data);
                            ^
    at async write (deno:ext/net/01_net.js:33:12)
    at async BufWriter.write (https://deno.land/[email protected]/io/bufio.ts:499:29)
    at async writeResponse (https://deno.land/[email protected]/http/_io.ts:273:15)
    at async ServerRequest.respond (https://deno.land/[email protected]/http/server.ts:89:7)

The whole thing is in a try-catch block but it does nothing.

[lino-levan]

I'm also experiencing this error. I'm just sending a large string occasionally and it crashes the server seemingly randomly in an uncatchable fashion. I'm starting to wonder if this isn't a problem with Deno and instead is a problem with something else?

[Mrmaxmeier]

Here's a small reproducer that doesn't depend on message sizes:

import { serve } from "https://deno.land/[email protected]/http/mod.ts";

const HTTP_PORT = 4321;

setTimeout(async () => {
  const c = await Deno.connect({ port: HTTP_PORT });
  const d = new TextEncoder().encode(`GET / HTTP/1.1\r
Connection: Upgrade\r
Upgrade: websocket\r
Sec-WebSocket-Key: dummy\r
Sec-WebSocket-Version: 13\r
\r\n\r\n`);
  await c.write(d);
  c.close()
}, 100);

function reqHandler(req: Request) {
  try {
    const { socket: ws, response } = Deno.upgradeWebSocket(req);
    ws.onopen = () => {
      try {
        // by the time we send our message the socket is closed, though
        // we have not noticed it yet and ws.readyState is still OPEN
        ws.send("something")
      } catch (e) {
        console.error(e)
      }
    }
    return response;
  } catch (e) {
    console.error(e)
    return new Response(null, { status: 500 })
  }
}

serve(reqHandler, { port: HTTP_PORT });

⋊> ~ deno run --allow-net repro.ts
Listening on http://localhost:4321/
Error 'IO error: Broken pipe (os error 32)' contains boxed error of unknown type:
  Io(Os { code: 32, kind: BrokenPipe, message: "Broken pipe" })
  Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }
error: Uncaught (in promise) Error: IO error: Broken pipe (os error 32)
⋊> ~ echo $status
1

I've encountered this with a Deno 1.25.0 server, but the snippet reproduces with 1.26 and current main as well.

[clarknova]

I run into the same issue with an seemingly uncatchable error: Uncaught (in promise) Error: IO error: Broken pipe (os error 32)

In React 18 useffect hooks run twice in strict mode. So connecting and immediately closing the connection is a common occurrence...

Edit: This only happens when I run it on osx. Works without issue in a docker container

[qkniep]

This seems to be solved in Deno's 1.27 release. At least the above reproduction code does no longer throw an error for me after upgrading from 1.25 to 1.27. For reference, I tried this on Manjaro Linux and Ubuntu 20.04.

[lino-levan]

Can confirm that the reproc no longer crashes. (M1, Mac OS 13)

I will see if I can reproduce the error outside of this.

Edit: I was able to reproduce the bug. If you take the first proposed reproc and change the Math.pow to 26, it will throw IO error. Wondering if this is intentional or not? If it is, it's really annoying.

 const content = new Array(Math.pow(2, 26)).join(":")

[irbull]

I'm not sure I'd feel confident if the errors just went away. This is clearly some timing / size issue in the underlying rust code / network layer. That happens, code's buggy, pipes break!

The fact that an underlying error cannot be caught by the calling code is concerning. If the calling code could catch these errors, then we could handle them in the application. I think reporting (and ultimately fixing these errors) should still be prioritized, but if underlying network problems leads to system crashes which cannot be avoided, then the platform is unusable.

[fucksophie]

I can reproduce with the code provided in this issue! This issue originally happened to me after I tried connecting 150 websocket connections to one server.

Version:

deno 1.31.1 (release, x86_64-unknown-linux-gnu)
v8 11.0.226.13
typescript 4.9.4

[alangecker]

I got the same error and am able to narrow it down to following code which reliably crashes opened with chromium, but not with firefox.

error: Uncaught (in promise) Error: IO error: Broken pipe (os error 32): Broken pipe (os error 32)

import { Application, Router } from 'https://deno.land/x/[email protected]/mod.ts'

const router = new Router()

router.get('/', ctx => {
  ctx.response.headers.set('content-type', 'text/html')
  ctx.response.body = `
  <!DOCTYPE html><script>
    const wsc = new WebSocket('ws://localhost:1111/ws', 'test')
    wsc.addEventListener('open', () => { console.log('connected') })
    wsc.addEventListener('error', (event) => { console.error(event) })
  </script>
  `
})
router.get('/ws', ctx => {
  try {
    const sock = ctx.upgrade();
    sock.onopen = () => {
      try {
        sock.send("{}")
        sock.send("{}")
      } catch(err) {
        console.log('onopen', err)
      }
    }
    sock.onerror = (e) => {
      console.log("socket errored:", e);
    }
    sock.onclose = () => {
      console.log("socket closed");
    }
  } catch(err) {
    console.log(err)
  }
});

const app = new Application()
app.use(router.routes());
app.use(router.allowedMethods())
app.listen({ port: 1111 })
console.log(`oak server running at http://localhost:1111/`)

• deno 1.31.1
• chromium 111.0.5563.64
• firefox 110.0.1

[littledivy]

Thanks for the report! This was fixed in the latest release (1.36.1). There was a race condition in the recv buffer. I'm going around closing issues related to this but please feel free to reopen if its still happening.

[soootaleb]

Wonderful @littledivy thanks a lot, you guys are building something awesome.

Can I ask you to share the commit that fixed the root cause please, I'm having a hard time finding a reference in the issues mentioned around here.

Edit: Is it this one ? https://github.com/denoland/fastwebsockets/commit/93d8aff85020fc95f8a654a97497ddfcd8822990

Regards

[littledivy]

Yes its that one (https://github.com/denoland/fastwebsockets/commit/93d8aff85020fc95f8a654a97497ddfcd8822990) and relavant commit in Deno: https://github.com/denoland/deno/commit/91dc6fa5f11f5621c21397c783ee899bfd9657ef