Subsequent 1KB+ responses on reused connection delayed by 40 milliseconds

[tdryer]

When running touche with a client that supports connection reuse on Linux, responses after the first response on the same connection that are larger than 1KB experience a 40 millisecond delay:

$ time curl -s http://localhost:4444 http://localhost:4444 > /dev/null
real 0m0.047s

This delay occurs due to an interaction between std::io::copy, Nagle's algorithm, and TCP delayed ACK:

• touche attempts to buffer writes using BufWriter. However, when a fixed-length response is larger than 1024 bytes, touche writes it using std::io::copy. The problem with this is that the implementation of copy has a specialization for BufWriter that allows it to reuse the writer's internal buffer space. In certain cases, this specialization appears to flush the buffer before and after performing the copy. This results in the response headers and body being sent in at least two separate TCP segments.
• TCP delayed ACK causes the client to wait before acknowledging the first segment, because it is smaller than the maximum segment size.
• Nagle's algorithm causes the server to wait before sending the second segment, because the first segment is unacknowledged.

I think the first response on a connection is unaffected because Linux always acknowledges the first data segment immediately.

This could be fixed in touche by either:

• Avoiding std::io::copy to ensure that the BufWriter is not flushed unexpectedly.
• Disabling Nagle's algorithm using TcpStream::set_nodelay.

[reu]

Thanks for the thoroughly investigation! I will take a better look into this too, but that was totally unexpected behavior. And yeah, maybe we should disable Nagle's algorithm by default and expose a flag if the user doesn't want that.

[reu]

Just letting you know that version 0.0.14 just add the option to disable Nagle's for every incoming connection to the server https://github.com/reu/touche/commit/8d1092f417a5f0c9db22204394020504b993549e.

I am still keeping this open though for now for more testing yet.