neqo icon indicating copy to clipboard operation
neqo copied to clipboard
verified publisher icon mozilla

refactor(bin/bench): be specific about up- and download size

Open mxinden opened this issue 8 months ago • 7 comments

neqo-bin has a benchmark for download throughput, requests per second, handshakes per second and upload throughput. Previously this was specified as:

struct Benchmark {
   name: String,
   requests: Vec<usize>,
   upload: bool,
}

This pull request makes the up- and download size explicit:

struct Benchmark {
    num_requests: usize,
    upload_size: usize,
    download_size: usize,
}

In addition, while previously one could either do an up- or a download benchmark, one could not do a combination of the two. This is due to the following line. Note that it ignores the URL response size parameter.

https://github.com/mozilla/neqo/blob/e72ae0ca8b3b413f854b136957c5d2bc83d69968/neqo-bin/src/server/http3.rs#L99-L102

This pull request fixes the above. When neqo-server receives a POST request, it waits for the client to close its write side, then responds with the amount of bytes specified in the URL path.

mxinden avatar Mar 31 '25 14:03 mxinden

Oh, good catch.

larseggert avatar Mar 31 '25 14:03 larseggert

Can we keep a bidirectional transfer as part of the benches?

larseggert avatar Mar 31 '25 14:03 larseggert

Can we keep a bidirectional transfer as part of the benches?

For sure. I will do a follow-up pull request. Sounds good?

mxinden avatar Mar 31 '25 14:03 mxinden

Failed Interop Tests

QUIC Interop Runner, client vs. server, differences relative to 847a5989f20d30eb3675d1435180fd48a8c6a3ad.

neqo-latest as client

neqo-latest as server

All results

Succeeded Interop Tests

QUIC Interop Runner, client vs. server

neqo-latest as client

neqo-latest as server

Unsupported Interop Tests

QUIC Interop Runner, client vs. server

neqo-latest as client

neqo-latest as server

github-actions[bot] avatar Mar 31 '25 14:03 github-actions[bot]

Benchmark results

Performance differences relative to 847a5989f20d30eb3675d1435180fd48a8c6a3ad.

1-conn/1-100mb-resp/mtu-1504 (aka. Download)/client: :broken_heart: Performance has regressed.
       time:   [720.29 ms 724.16 ms 728.03 ms]
       thrpt:  [137.36 MiB/s 138.09 MiB/s 138.83 MiB/s]
change:
       time:   [+2.1637% +2.9787% +3.8215%] (p = 0.00 -2.8926% -2.1179%]
1-conn/10_000-parallel-1b-resp/mtu-1504 (aka. RPS)/client: No change in performance detected.
       time:   [352.01 ms 353.40 ms 354.77 ms]
       thrpt:  [28.187 Kelem/s 28.297 Kelem/s 28.408 Kelem/s]
change:
       time:   [-0.4056% +0.1571% +0.7741%] (p = 0.60 > 0.05)
       thrpt:  [-0.7682% -0.1568% +0.4073%]
1-conn/1-1b-resp/mtu-1504 (aka. HPS)/client: No change in performance detected.
       time:   [25.511 ms 25.669 ms 25.833 ms]
       thrpt:  [38.711   B/s 38.958   B/s 39.199   B/s]
change:
       time:   [-1.2032% -0.3841% +0.5317%] (p = 0.39 > 0.05)
       thrpt:  [-0.5289% +0.3856% +1.2179%]

Found 2 outliers among 100 measurements (2.00%)
2 (2.00%) high mild
1-conn/1-100mb-req/mtu-1504 (aka. Upload)/client: :green_heart: Performance has improved.
       time:   [1.8498 s 1.8652 s 1.8820 s]
       thrpt:  [53.135 MiB/s 53.612 MiB/s 54.060 MiB/s]
change:
       time:   [-5.6540% -3.9480% -2.3320%] (p = 0.00 +4.1102% +5.9928%]

Found 2 outliers among 100 measurements (2.00%)
2 (2.00%) high severe
decode 4096 bytes, mask ff: No change in performance detected.
       time:   [12.077 µs 12.112 µs 12.155 µs]
       change: [-0.1648% +0.2954% +0.7917%] (p = 0.28 > 0.05)

Found 15 outliers among 100 measurements (15.00%)
3 (3.00%) low severe
3 (3.00%) low mild
1 (1.00%) high mild
8 (8.00%) high severe
decode 1048576 bytes, mask ff: No change in performance detected.
       time:   [3.1342 ms 3.1456 ms 3.1586 ms]
       change: [-0.4604% +0.0720% +0.5650%] (p = 0.78 > 0.05)

Found 12 outliers among 100 measurements (12.00%)
12 (12.00%) high severe
decode 4096 bytes, mask 7f: No change in performance detected.
       time:   [20.171 µs 20.233 µs 20.303 µs]
       change: [-0.2690% +0.1039% +0.5710%] (p = 0.64 > 0.05)

Found 19 outliers among 100 measurements (19.00%)
1 (1.00%) low severe
1 (1.00%) low mild
2 (2.00%) high mild
15 (15.00%) high severe
decode 1048576 bytes, mask 7f: No change in performance detected.
       time:   [5.2481 ms 5.2598 ms 5.2731 ms]
       change: [-0.3391% -0.0069% +0.3347%] (p = 0.95 > 0.05)

Found 14 outliers among 100 measurements (14.00%)
14 (14.00%) high severe
decode 4096 bytes, mask 3f: No change in performance detected.
       time:   [7.0138 µs 7.0420 µs 7.0771 µs]
       change: [-0.7778% -0.0870% +0.7085%] (p = 0.82 > 0.05)

Found 15 outliers among 100 measurements (15.00%)
2 (2.00%) low severe
1 (1.00%) low mild
1 (1.00%) high mild
11 (11.00%) high severe
decode 1048576 bytes, mask 3f: No change in performance detected.
       time:   [1.7949 ms 1.8020 ms 1.8105 ms]
       change: [-0.4707% +0.1434% +0.6976%] (p = 0.66 > 0.05)

Found 10 outliers among 100 measurements (10.00%)
1 (1.00%) high mild
9 (9.00%) high severe
1000 streams of 1 bytes/multistream: Change within noise threshold.
       time:   [25.430 ms 25.457 ms 25.484 ms]
       change: [-0.6185% -0.4790% -0.3355%] (p = 0.00 Found 43 outliers among 500 measurements (8.60%)
42 (8.40%) high mild
1 (0.20%) high severe
1000 streams of 1000 bytes/multistream: Change within noise threshold.
       time:   [143.98 ms 144.02 ms 144.05 ms]
       change: [+0.0569% +0.0943% +0.1299%] (p = 0.00 Found 3 outliers among 500 measurements (0.60%)
3 (0.60%) high mild
coalesce_acked_from_zero 1+1 entries: No change in performance detected.
       time:   [94.710 ns 95.020 ns 95.335 ns]
       change: [-0.4107% +0.0504% +0.4639%] (p = 0.84 > 0.05)

Found 13 outliers among 100 measurements (13.00%)
1 (1.00%) low mild
9 (9.00%) high mild
3 (3.00%) high severe
coalesce_acked_from_zero 3+1 entries: No change in performance detected.
       time:   [112.57 ns 112.95 ns 113.38 ns]
       change: [-0.5902% -0.0812% +0.4672%] (p = 0.77 > 0.05)

Found 14 outliers among 100 measurements (14.00%)
2 (2.00%) low mild
1 (1.00%) high mild
11 (11.00%) high severe
coalesce_acked_from_zero 10+1 entries: No change in performance detected.
       time:   [112.05 ns 112.50 ns 113.04 ns]
       change: [-0.7077% -0.1476% +0.3912%] (p = 0.60 > 0.05)

Found 16 outliers among 100 measurements (16.00%)
4 (4.00%) low severe
3 (3.00%) low mild
2 (2.00%) high mild
7 (7.00%) high severe
coalesce_acked_from_zero 1000+1 entries: No change in performance detected.
       time:   [93.471 ns 93.910 ns 94.383 ns]
       change: [-0.0664% +1.2968% +2.6933%] (p = 0.07 > 0.05)

Found 8 outliers among 100 measurements (8.00%)
4 (4.00%) high mild
4 (4.00%) high severe
RxStreamOrderer::inbound_frame(): Change within noise threshold.
       time:   [116.68 ms 116.74 ms 116.79 ms]
       change: [-0.1755% -0.1150% -0.0510%] (p = 0.00 Found 18 outliers among 100 measurements (18.00%)
3 (3.00%) low severe
6 (6.00%) low mild
6 (6.00%) high mild
3 (3.00%) high severe
SentPackets::take_ranges: No change in performance detected.
       time:   [8.3213 µs 8.5454 µs 8.7472 µs]
       change: [-4.2212% -1.5163% +1.4174%] (p = 0.32 > 0.05)

Found 23 outliers among 100 measurements (23.00%)
8 (8.00%) low severe
11 (11.00%) low mild
3 (3.00%) high mild
1 (1.00%) high severe
transfer/pacing-false/varying-seeds: Change within noise threshold.
       time:   [35.837 ms 35.907 ms 35.976 ms]
       change: [+0.8034% +1.0614% +1.3195%] (p = 0.00 Found 2 outliers among 100 measurements (2.00%)
2 (2.00%) low mild
transfer/pacing-true/varying-seeds: No change in performance detected.
       time:   [36.535 ms 36.647 ms 36.757 ms]
       change: [-0.2275% +0.2178% +0.6551%] (p = 0.33 > 0.05)

Found 3 outliers among 100 measurements (3.00%)
3 (3.00%) high mild
transfer/pacing-false/same-seed: Change within noise threshold.
       time:   [35.514 ms 35.558 ms 35.601 ms]
       change: [+0.6000% +0.7908% +0.9695%] (p = 0.00 Found 6 outliers among 100 measurements (6.00%)
5 (5.00%) low mild
1 (1.00%) high mild
transfer/pacing-true/same-seed: Change within noise threshold.
       time:   [37.447 ms 37.521 ms 37.594 ms]
       change: [+0.4199% +0.6821% +0.9570%] (p = 0.00 Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) high mild

Client/server transfer results

Performance differences relative to 847a5989f20d30eb3675d1435180fd48a8c6a3ad.

Transfer of 33554432 bytes over loopback, 30 runs. All unit-less numbers are in milliseconds.

Client Server CC Pacing Mean ± σ Min Max MiB/s ± σ Δ main Δ main
neqo neqo reno on 336.9 ± 56.5 298.4 530.8 95.0 ± 0.6 3.6 1.1%
neqo neqo reno 334.9 ± 93.8 296.2 814.9 95.6 ± 0.3 -20.6 -5.8%
neqo neqo cubic on 327.7 ± 37.8 299.5 508.9 97.6 ± 0.8 5.6 1.7%
neqo neqo cubic 325.7 ± 22.0 296.7 386.2 98.2 ± 1.5 -4.8 -1.5%
google neqo reno on 770.0 ± 93.0 566.3 986.8 41.6 ± 0.3 8.3 1.1%
google neqo reno 774.6 ± 99.1 550.7 997.2 41.3 ± 0.3 14.9 2.0%
google neqo cubic on 764.3 ± 84.6 574.0 912.7 41.9 ± 0.4 8.9 1.2%
google neqo cubic 767.4 ± 87.4 570.6 977.7 41.7 ± 0.4 5.3 0.7%
google google 576.0 ± 35.0 552.5 730.8 55.6 ± 0.9 3.2 0.6%
neqo msquic reno on 267.4 ± 21.7 245.8 334.0 119.7 ± 1.5 -1.3 -0.5%
neqo msquic reno 274.3 ± 46.2 241.6 441.0 116.7 ± 0.7 0.5 0.2%
neqo msquic cubic on 278.1 ± 45.2 241.9 421.3 115.1 ± 0.7 14.7 5.6%
neqo msquic cubic 272.7 ± 37.1 241.7 427.0 117.3 ± 0.9 5.3 2.0%
msquic msquic 193.8 ± 27.2 161.5 291.4 165.1 ± 1.2 -2.9 -1.5%

:arrow_down: Download logs

github-actions[bot] avatar Mar 31 '25 15:03 github-actions[bot]

I was wrong. Updated the PR description. Still worth merging.

While a bug, this does not fix #2538, given that the HTTP3 server would simply ignore the /104857600 URL path on receiving a POST.

https://github.com/mozilla/neqo/blob/e72ae0ca8b3b413f854b136957c5d2bc83d69968/neqo-bin/src/server/http3.rs#L99-L102

mxinden avatar Mar 31 '25 18:03 mxinden

Can we add separate options for upload and download size, so that the request URL contains the download size and the client just uploads/POSTs the upload size?

martinthomson avatar Apr 01 '25 01:04 martinthomson

@mxinden is this OBE with #2599 merged?

larseggert avatar Jul 03 '25 15:07 larseggert

Closing here in favor of https://github.com/mozilla/neqo/issues/2792.

mxinden avatar Jul 11 '25 11:07 mxinden