Performance issue on internet

[stormshield-damiend]

This is similar to what discussed in #2153 .

Here are some tests we made on the following setup :

• Two VMs in two different datacenters in the cloud
• Latency between them is around 10ms (ping)
• Network driver is virtio_net with generic-segmentation-offload: on, generic-receive-offload: on and tx-udp-segmentation: off [fixed] (from ethtool)
• We test uploading from one VM to the other

First here is the result of an iperf3 test between the two VMs (TCP non ciphered)

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.12 GBytes   958 Mbits/sec  1102             sender
[  5]   0.00-10.01  sec  1.11 GBytes   955 Mbits/sec                  receiver

This matches the expected line rate between the two VMs.

We then use secnetperf from msquic project (UDP with encryption) https://github.com/microsoft/msquic/tree/main/src/perf

Result: Upload 1298923520 bytes @ 913880 kbps (11370.619 ms).

We are close to the line rate.

We then used quinn/perf client and server (UDP with encryption)

Overall stats:
RPS: 37.25 (373 requests in 10.01s)

Stream metrics:

      │ Upload Duration │ Download Duration | FBL        | Upload Throughput | Download Throughput
──────┼─────────────────┼───────────────────┼────────────┼───────────────────┼────────────────────
 AVG  │         26.71ms │           57.00µs │    57.00µs │       323.99 Mb/s │          0.00 Mb/s
 P0   │         18.21ms │            1.00µs │     1.00µs │        99.09 Mb/s │          0.00 Mb/s
 P10  │         22.98ms │            2.00µs │     2.00µs │       287.31 Mb/s │          0.00 Mb/s
 P50  │         25.30ms │           47.00µs │    47.00µs │       331.87 Mb/s │          0.00 Mb/s
 P90  │         29.20ms │          124.00µs │   124.00µs │       365.43 Mb/s │          0.00 Mb/s
 P100 │         84.67ms │          398.00µs │   398.00µs │       460.59 Mb/s │          0.00 Mb/s

We are around half the line rate.

If we disable crypto using --no-protection

Overall stats:
RPS: 38.84 (389 requests in 10.02s)

Stream metrics:

      │ Upload Duration │ Download Duration | FBL        | Upload Throughput | Download Throughput
──────┼─────────────────┼───────────────────┼────────────┼───────────────────┼────────────────────
 AVG  │         25.58ms │           76.00µs │    76.00µs │       337.91 Mb/s │          0.00 Mb/s
 P0   │         15.38ms │            1.00µs │     1.00µs │       100.66 Mb/s │          0.00 Mb/s
 P10  │         21.22ms │            5.00µs │     4.00µs │       283.64 Mb/s │          0.00 Mb/s
 P50  │         24.70ms │           70.00µs │    70.00µs │       339.74 Mb/s │          0.00 Mb/s
 P90  │         29.60ms │          143.00µs │   143.00µs │       395.84 Mb/s │          0.00 Mb/s
 P100 │         83.33ms │          578.00µs │   578.00µs │       545.26 Mb/s │          0.00 Mb/s

We get a bit better but are still far from the line rate

I can dedicate some work time to work on this issue, if you have any insight feel free to comment.