Throughput performance degrades significantly when I greatly increase buffer size

[jiridanek]

https://github.com/gpjt/rsp/blob/9d1b40c69a05cc784b78c2386efa2afcce899b8c/src/connection.c#L16-L18

When I use

const int BUFFER_SIZE = 8 * 1024;

I still get good performance

but when I increase it to 16k or 32k, I get very bad behavior

const int BUFFER_SIZE = 32 * 1024;

Benchmark

I am using iperf3 as a server, that listens on 5201

iperf3 -s

and I connect to the epoll proxy with iperf3 client

iperf3 -c 127.0.0.1 -p 5101 -Z

Good result

$ iperf3 -c 127.0.0.1 -p 5101 -Z
Connecting to host 127.0.0.1, port 5101
[  5] local 127.0.0.1 port 59662 connected to 127.0.0.1 port 5101
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  2.91 GBytes  25.0 Gbits/sec    0   2.62 MBytes       
[  5]   1.00-2.00   sec  2.85 GBytes  24.5 Gbits/sec    0   2.62 MBytes       
[  5]   2.00-3.00   sec  2.97 GBytes  25.5 Gbits/sec    0   2.62 MBytes       
[  5]   3.00-4.00   sec  2.84 GBytes  24.4 Gbits/sec    0   2.62 MBytes       
[  5]   4.00-5.00   sec  2.97 GBytes  25.5 Gbits/sec    0   2.62 MBytes       
[  5]   5.00-6.00   sec  2.97 GBytes  25.5 Gbits/sec    0   2.62 MBytes       
[  5]   6.00-7.00   sec  2.96 GBytes  25.4 Gbits/sec    0   2.62 MBytes       
[  5]   7.00-8.00   sec  2.93 GBytes  25.2 Gbits/sec    0   2.62 MBytes       
[  5]   8.00-9.00   sec  2.93 GBytes  25.2 Gbits/sec    0   2.62 MBytes       
[  5]   9.00-10.00  sec  3.06 GBytes  26.3 Gbits/sec    0   2.62 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  29.4 GBytes  25.2 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  29.4 GBytes  25.2 Gbits/sec                  receiver

iperf Done.

Bad result

$ iperf3 -c 127.0.0.1 -p 5101 -Z
Connecting to host 127.0.0.1, port 5101
[  5] local 127.0.0.1 port 55962 connected to 127.0.0.1 port 5101
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  3.31 GBytes  28.5 Gbits/sec    0   1.56 MBytes       
[  5]   1.00-2.00   sec  3.35 GBytes  28.8 Gbits/sec    0   1.56 MBytes       
[  5]   2.00-3.00   sec  3.20 GBytes  27.5 Gbits/sec    0   1.56 MBytes       
[  5]   3.00-4.00   sec   682 MBytes  5.73 Gbits/sec    0   1.56 MBytes       
[  5]   4.00-5.00   sec  67.5 MBytes   566 Mbits/sec    0   1.56 MBytes       
[  5]   5.00-6.00   sec  50.0 MBytes   419 Mbits/sec    0   1.56 MBytes       
[  5]   6.00-7.00   sec  41.2 MBytes   346 Mbits/sec    0   1.56 MBytes       
[  5]   7.00-8.00   sec  37.5 MBytes   315 Mbits/sec    0   1.56 MBytes       
[  5]   8.00-9.00   sec  31.2 MBytes   262 Mbits/sec    0   1.56 MBytes       
[  5]   9.00-10.00  sec  27.5 MBytes   231 Mbits/sec    0   1.56 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  10.8 GBytes  9.26 Gbits/sec    0             sender
[  5]   0.00-10.43  sec  10.8 GBytes  8.88 Gbits/sec                  receiver

iperf Done.