neqo icon indicating copy to clipboard operation
neqo copied to clipboard
verified publisher icon mozilla

refactor(bin/bench): make up- and download size explicit

Open mxinden opened this issue 7 months ago • 2 comments

neqo-bin has a benchmark for download throughput, requests per second, handshakes per second and upload throughput. Previously this was specified the following way, where the usize in requests could be either the up- or download size.

struct Benchmark {
   name: String,
   requests: Vec<usize>,
   upload: bool,
}

This commit makes the up- and download size explicit:

struct Benchmark {
    num_requests: usize,
    upload_size: usize,
    download_size: usize,
}

Extracted out of https://github.com/mozilla/neqo/pull/2546.

mxinden avatar Apr 27 '25 13:04 mxinden

Failed Interop Tests

QUIC Interop Runner, client vs. server, differences relative to d35049e9e4e51d8c272a5e588a88abb2bbb7a628.

neqo-latest as client

neqo-latest as server

All results

Succeeded Interop Tests

QUIC Interop Runner, client vs. server

neqo-latest as client

  • neqo-latest vs. aioquic: :warning:DC :rocket:~~H~~ LR :warning:C20 :rocket:~~M~~ S R Z :rocket:~~3~~ B :rocket:~~U~~ L2 C1 :warning:C2 6 V2 BP BA
  • neqo-latest vs. go-x-net: H :rocket:~~DC~~ LR :rocket:~~M B~~ U A L2 C2 :rocket:~~6~~
  • neqo-latest vs. haproxy: :warning:DC LR :rocket:~~M~~ S :rocket:~~R~~ Z :warning:3 U A :warning:L1 C2 V2 :rocket:~~C1 6~~
  • neqo-latest vs. kwik: H DC LR C20 M S :warning:R Z :warning:3 U :rocket:~~B~~ A :rocket:~~C2~~ 6 V2
  • neqo-latest vs. linuxquic: H DC :warning:LR :rocket:~~C20 M Z~~ 3 :rocket:~~B U~~ E A L2 C2 :warning:6 V2 :warning:BP CM
  • neqo-latest vs. lsquic: H DC :warning:B :rocket:~~LR C20 M S R 3 U~~ E A L2 :warning:C2 6 BA :rocket:~~V2 BP~~ CM
  • neqo-latest vs. msquic: :warning:H DC LR M :warning:R :rocket:~~S~~ B U L2 C2 6 :rocket:~~V2~~ BP BA
  • neqo-latest vs. mvfst: H DC LR :rocket:~~M~~ R Z :warning:B :rocket:~~3~~ U :rocket:~~C2~~ BP BA
  • neqo-latest vs. neqo: H LR C20 S R Z :rocket:~~3~~ B U E :warning:L2 :rocket:~~L1~~ C1 C2 6 V2 BP BA :rocket:~~CM~~
  • neqo-latest vs. neqo-latest: H DC :warning:C20 M :rocket:~~LR~~ S :warning:R Z 3 B :rocket:~~U~~ E L1 :warning:L2 C2 6 :rocket:~~V2~~ BP BA CM
  • neqo-latest vs. nginx: :warning:DC :rocket:~~H~~ LR :rocket:~~C20 M S R Z 3~~ B :warning:A L1 C1 :warning:6 :rocket:~~C2~~
  • neqo-latest vs. ngtcp2: H DC :warning:C20 :rocket:~~LR~~ S R :rocket:~~Z~~ 3 :warning:U E A :rocket:~~L1~~ L2 C1 :rocket:~~C2 6~~ V2 :warning:BA :rocket:~~BP~~
  • neqo-latest vs. picoquic: H DC LR C20 :rocket:~~M~~ S :warning:R :rocket:~~3 B~~ U E :rocket:~~L1~~ L2 :rocket:~~C1~~ C2 :warning:6 V2 :rocket:~~BP BA~~
  • neqo-latest vs. quic-go: H :warning:DC LR M :rocket:~~C20~~ S R :warning:Z 3 :rocket:~~U~~ L1 L2 C1 C2 6 :warning:BP
  • neqo-latest vs. quiche: :rocket:~~H~~ DC :rocket:~~LR~~ C20 :warning:M S R Z :warning:3 B :warning:U :rocket:~~A~~ L2 :rocket:~~C1~~ C2 6
  • neqo-latest vs. quinn: :warning:H DC LR C20 M :warning:R Z :rocket:~~S 3 B~~ U E :warning:L1 L2 :warning:C1 C2 6 BP :rocket:~~BA~~
  • neqo-latest vs. s2n-quic: H :warning:DC LR B U E A L1 :warning:C1 C2 :rocket:~~L2~~ 6 BP
  • neqo-latest vs. tquic: :rocket:~~H DC~~ LR C20 :warning:M R :warning:3 B :rocket:~~Z U~~ A :warning:L1 L2 C1 :warning:C2 6
  • neqo-latest vs. xquic: H LR C20 M :warning:R :rocket:~~Z~~ 3 :rocket:~~B~~ U L2 :warning:6 BA :rocket:~~C2 BP~~

neqo-latest as server

Unsupported Interop Tests

QUIC Interop Runner, client vs. server

neqo-latest as client

neqo-latest as server

github-actions[bot] avatar Apr 27 '25 14:04 github-actions[bot]

Benchmark results

Performance differences relative to 95f9bedb40bc852f5f62b611ad6b2fd22c636843.

1-conn/1-100mb-resp/mtu-1504 (aka. Download)/client: :green_heart: Performance has improved.
       time:   [642.82 ms 643.79 ms 644.85 ms]
       thrpt:  [155.08 MiB/s 155.33 MiB/s 155.56 MiB/s]
change:
       time:   [−4.7881% −4.6127% −4.4062%] (p = 0.00 +4.8357% +5.0288%]

Found 5 outliers among 100 measurements (5.00%)
3 (3.00%) high mild
2 (2.00%) high severe
1-conn/10_000-parallel-1b-resp/mtu-1504 (aka. RPS)/client: Change within noise threshold.
       time:   [293.44 ms 294.76 ms 296.09 ms]
       thrpt:  [33.773 Kelem/s 33.926 Kelem/s 34.078 Kelem/s]
change:
       time:   [−1.4491% −0.7832% −0.1422%] (p = 0.02 +0.7894% +1.4705%]
1-conn/1-1b-resp/mtu-1504 (aka. HPS)/client: No change in performance detected.
       time:   [27.245 ms 27.403 ms 27.582 ms]
       thrpt:  [36.256   B/s 36.493   B/s 36.704   B/s]
change:
       time:   [−0.1205% +0.6917% +1.5132%] (p = 0.10 > 0.05)
       thrpt:  [−1.4907% −0.6870% +0.1206%]

Found 9 outliers among 100 measurements (9.00%)
1 (1.00%) high mild
8 (8.00%) high severe
1-conn/1-100mb-req/mtu-1504 (aka. Upload)/client: No change in performance detected.
       time:   [664.12 ms 678.57 ms 694.55 ms]
       thrpt:  [143.98 MiB/s 147.37 MiB/s 150.57 MiB/s]
change:
       time:   [−5.9733% −3.0824% +0.0083%] (p = 0.05 > 0.05)
       thrpt:  [−0.0083% +3.1805% +6.3528%]

Found 21 outliers among 100 measurements (21.00%)
3 (3.00%) high mild
18 (18.00%) high severe
decode 4096 bytes, mask ff: No change in performance detected.
       time:   [11.803 µs 11.841 µs 11.884 µs]
       change: [−0.4158% −0.0208% +0.4036%] (p = 0.92 > 0.05)

Found 16 outliers among 100 measurements (16.00%)
1 (1.00%) low severe
2 (2.00%) low mild
3 (3.00%) high mild
10 (10.00%) high severe
decode 1048576 bytes, mask ff: No change in performance detected.
       time:   [3.0227 ms 3.0321 ms 3.0433 ms]
       change: [−0.3183% +0.1154% +0.5888%] (p = 0.64 > 0.05)

Found 9 outliers among 100 measurements (9.00%)
9 (9.00%) high severe
decode 4096 bytes, mask 7f: No change in performance detected.
       time:   [19.998 µs 20.156 µs 20.397 µs]
       change: [−0.1328% +0.4427% +1.1736%] (p = 0.17 > 0.05)

Found 21 outliers among 100 measurements (21.00%)
5 (5.00%) low mild
16 (16.00%) high severe
decode 1048576 bytes, mask 7f: No change in performance detected.
       time:   [5.0384 ms 5.0501 ms 5.0632 ms]
       change: [−0.2857% +0.0516% +0.4196%] (p = 0.77 > 0.05)

Found 16 outliers among 100 measurements (16.00%)
1 (1.00%) high mild
15 (15.00%) high severe
decode 4096 bytes, mask 3f: No change in performance detected.
       time:   [8.2686 µs 8.3062 µs 8.3478 µs]
       change: [−0.3858% +0.3326% +1.1273%] (p = 0.46 > 0.05)

Found 22 outliers among 100 measurements (22.00%)
7 (7.00%) low mild
4 (4.00%) high mild
11 (11.00%) high severe
decode 1048576 bytes, mask 3f: No change in performance detected.
       time:   [1.5851 ms 1.5893 ms 1.5950 ms]
       change: [−0.6158% −0.0976% +0.4220%] (p = 0.69 > 0.05)

Found 4 outliers among 100 measurements (4.00%)
4 (4.00%) high severe
1000 streams of 1 bytes/multistream: No change in performance detected.
       time:   [29.563 ns 29.865 ns 30.171 ns]
       change: [−2.5699% −0.8784% +0.6509%] (p = 0.30 > 0.05)

Found 4 outliers among 500 measurements (0.80%)
4 (0.80%) high mild
1000 streams of 1000 bytes/multistream: :green_heart: Performance has improved.
       time:   [29.362 ns 29.668 ns 29.978 ns]
       change: [−10.523% −9.0441% −7.5917%] (p = 0.00 Found 4 outliers among 500 measurements (0.80%)
4 (0.80%) high mild
coalesce_acked_from_zero 1+1 entries: No change in performance detected.
       time:   [87.917 ns 88.204 ns 88.492 ns]
       change: [−0.4363% −0.0451% +0.3614%] (p = 0.83 > 0.05)

Found 8 outliers among 100 measurements (8.00%)
6 (6.00%) high mild
2 (2.00%) high severe
coalesce_acked_from_zero 3+1 entries: No change in performance detected.
       time:   [105.50 ns 105.83 ns 106.17 ns]
       change: [−0.3745% +0.0182% +0.4164%] (p = 0.93 > 0.05)

Found 19 outliers among 100 measurements (19.00%)
2 (2.00%) low mild
8 (8.00%) high mild
9 (9.00%) high severe
coalesce_acked_from_zero 10+1 entries: No change in performance detected.
       time:   [105.07 ns 105.40 ns 105.83 ns]
       change: [−0.3783% +0.0971% +0.6213%] (p = 0.72 > 0.05)

Found 16 outliers among 100 measurements (16.00%)
3 (3.00%) low severe
4 (4.00%) low mild
1 (1.00%) high mild
8 (8.00%) high severe
coalesce_acked_from_zero 1000+1 entries: No change in performance detected.
       time:   [88.600 ns 88.729 ns 88.879 ns]
       change: [−0.5760% +0.2721% +1.2984%] (p = 0.59 > 0.05)

Found 11 outliers among 100 measurements (11.00%)
5 (5.00%) high mild
6 (6.00%) high severe
RxStreamOrderer::inbound_frame(): Change within noise threshold.
       time:   [107.10 ms 107.17 ms 107.24 ms]
       change: [−1.3388% −1.0870% −0.9068%] (p = 0.00 Found 20 outliers among 100 measurements (20.00%)
2 (2.00%) low severe
9 (9.00%) low mild
8 (8.00%) high mild
1 (1.00%) high severe
sent::Packets::take_ranges: No change in performance detected.
       time:   [8.0801 µs 8.2870 µs 8.4799 µs]
       change: [−1.5019% +5.1452% +15.062%] (p = 0.30 > 0.05)

Found 22 outliers among 100 measurements (22.00%)
3 (3.00%) low severe
13 (13.00%) low mild
3 (3.00%) high mild
3 (3.00%) high severe
transfer/pacing-false/varying-seeds: Change within noise threshold.
       time:   [34.979 ms 35.052 ms 35.126 ms]
       change: [−1.0670% −0.7732% −0.4808%] (p = 0.00 Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) high mild
transfer/pacing-true/varying-seeds: Change within noise threshold.
       time:   [35.968 ms 36.067 ms 36.166 ms]
       change: [+1.0575% +1.4853% +1.8918%] (p = 0.00 Found 2 outliers among 100 measurements (2.00%)
2 (2.00%) high mild
transfer/pacing-false/same-seed: Change within noise threshold.
       time:   [34.962 ms 35.024 ms 35.089 ms]
       change: [+1.0269% +1.2692% +1.5048%] (p = 0.00 Found 2 outliers among 100 measurements (2.00%)
1 (1.00%) low mild
1 (1.00%) high severe
transfer/pacing-true/same-seed: Change within noise threshold.
       time:   [36.324 ms 36.378 ms 36.432 ms]
       change: [−0.7901% −0.5390% −0.2845%] (p = 0.00 Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) high mild

Client/server transfer results

Performance differences relative to 95f9bedb40bc852f5f62b611ad6b2fd22c636843.

Transfer of 33554432 bytes over loopback, min. 100 runs. All unit-less numbers are in milliseconds.

Client vs. server (params) Mean ± σ Min Max MiB/s ± σ Δ main Δ main
google vs. google 451.9 ± 4.6 444.7 468.3 70.8 ± 7.0
google vs. neqo (cubic, paced) 317.4 ± 4.2 308.9 332.8 100.8 ± 7.6 :green_heart: -1.7 -0.5%
msquic vs. msquic 126.6 ± 19.6 107.3 217.3 252.8 ± 1.6
msquic vs. neqo (cubic, paced) 274.7 ± 42.7 234.3 483.0 116.5 ± 0.7 7.1 2.7%
neqo vs. google (cubic, paced) 751.1 ± 7.1 743.7 805.8 42.6 ± 4.5 1.0 0.1%
neqo vs. msquic (cubic, paced) 155.7 ± 5.7 147.2 182.4 205.5 ± 5.6 -0.2 -0.1%
neqo vs. neqo (cubic) 210.0 ± 4.2 200.0 219.1 152.4 ± 7.6 :green_heart: -1.6 -0.7%
neqo vs. neqo (cubic, paced) 210.7 ± 4.9 200.7 231.3 151.9 ± 6.5 -1.0 -0.5%
neqo vs. neqo (reno) 208.5 ± 4.0 202.8 215.7 153.5 ± 8.0 0.3 0.2%
neqo vs. neqo (reno, paced) 211.2 ± 4.2 201.6 221.5 151.5 ± 7.6 :broken_heart: 1.9 0.9%
neqo vs. quiche (cubic, paced) 191.7 ± 5.0 184.5 212.5 167.0 ± 6.4 -0.5 -0.3%
neqo vs. s2n (cubic, paced) 219.8 ± 3.9 213.5 228.8 145.6 ± 8.2 -0.5 -0.2%
quiche vs. neqo (cubic, paced) 626.0 ± 128.1 433.0 919.6 51.1 ± 0.2 14.9 2.4%
quiche vs. quiche 145.2 ± 4.9 137.7 159.6 220.3 ± 6.5
s2n vs. neqo (cubic, paced) 299.5 ± 12.9 274.6 331.9 106.9 ± 2.5 -0.7 -0.2%
s2n vs. s2n 246.2 ± 22.4 233.2 348.2 130.0 ± 1.4

Download data for profiler.firefox.com or download performance comparison data.

github-actions[bot] avatar Apr 27 '25 14:04 github-actions[bot]

🐰 Bencher Report

Branchbin-bench-refactor
Testbedt-linux64-ms-280
Click to view all benchmark results
BenchmarkLatencyBenchmark Result
nanoseconds (ns)
(Result Δ%)		Upper Boundary
nanoseconds (ns)
(Limit %)
1-conn/1-100mb-req/mtu-1504 (aka. Upload)/client📈 view plot
🚷 view threshold		665,910,000.00 ns
(+0.19%)Baseline: 664,657,313.43 ns
735,073,273.13 ns
(90.59%)
1-conn/1-100mb-resp/mtu-1504 (aka. Download)/client📈 view plot
🚷 view threshold		27,157,000.00 ns
(-95.62%)Baseline: 620,060,552.24 ns
892,872,644.87 ns
(3.04%)
1-conn/1-1b-resp/mtu-1504 (aka. HPS)/client📈 view plot
🚷 view threshold		27,040,000.00 ns
(-0.61%)Baseline: 27,204,910.45 ns
27,684,702.43 ns
(97.67%)
1-conn/10_000-parallel-1b-resp/mtu-1504 (aka. RPS)/client📈 view plot
🚷 view threshold		303,270,000.00 ns
(-0.52%)Baseline: 304,859,850.75 ns
316,527,094.07 ns
(95.81%)
1000 streams of 1 bytes/multistream📈 view plot
🚷 view threshold		31.54 ns
(-17.04%)Baseline: 38.02 ns
55.25 ns
(57.08%)
1000 streams of 1000 bytes/multistream📈 view plot
🚷 view threshold		31.53 ns
(-15.92%)Baseline: 37.50 ns
54.92 ns
(57.42%)
RxStreamOrderer::inbound_frame()📈 view plot
🚷 view threshold		111,200,000.00 ns
(+0.66%)Baseline: 110,466,134.33 ns
114,729,097.56 ns
(96.92%)
SentPackets::take_ranges📈 view plot
🚷 view threshold		8,108.70 ns
(+1.77%)Baseline: 7,968.04 ns
8,166.50 ns
(99.29%)
coalesce_acked_from_zero 1+1 entries📈 view plot
🚷 view threshold		88.58 ns
(-0.13%)Baseline: 88.69 ns
89.31 ns
(99.18%)
coalesce_acked_from_zero 10+1 entries📈 view plot
🚷 view threshold		105.64 ns
(-0.29%)Baseline: 105.95 ns
106.93 ns
(98.80%)
coalesce_acked_from_zero 1000+1 entries📈 view plot
🚷 view threshold		88.85 ns
(-0.49%)Baseline: 89.29 ns
91.62 ns
(96.98%)
coalesce_acked_from_zero 3+1 entries📈 view plot
🚷 view threshold		106.23 ns
(-0.29%)Baseline: 106.54 ns
107.45 ns
(98.87%)
decode 1048576 bytes, mask 3f📈 view plot
🚷 view threshold		1,590,600.00 ns
(-1.75%)Baseline: 1,618,883.58 ns
1,773,266.47 ns
(89.70%)
decode 1048576 bytes, mask 7f📈 view plot
🚷 view threshold		5,062,200.00 ns
(-0.04%)Baseline: 5,064,050.75 ns
5,092,197.90 ns
(99.41%)
decode 1048576 bytes, mask ff📈 view plot
🚷 view threshold		3,029,900.00 ns
(-0.21%)Baseline: 3,036,223.88 ns
3,069,542.14 ns
(98.71%)
decode 4096 bytes, mask 3f📈 view plot
🚷 view threshold		8,286.90 ns
(+5.09%)Baseline: 7,885.75 ns
10,224.81 ns
(81.05%)
decode 4096 bytes, mask 7f📈 view plot
🚷 view threshold		19,990.00 ns
(+0.41%)Baseline: 19,909.09 ns
20,415.00 ns
(97.92%)
decode 4096 bytes, mask ff📈 view plot
🚷 view threshold		11,815.00 ns
(+0.01%)Baseline: 11,813.73 ns
11,980.31 ns
(98.62%)
transfer/pacing-false/same-seed📈 view plot
🚷 view threshold		34,869,000.00 ns
(+0.01%)Baseline: 34,865,880.60 ns
36,540,629.91 ns
(95.43%)
transfer/pacing-false/varying-seeds📈 view plot
🚷 view threshold		35,060,000.00 ns
(+0.14%)Baseline: 35,009,447.76 ns
36,742,620.43 ns
(95.42%)
transfer/pacing-true/same-seed📈 view plot
🚷 view threshold		36,067,000.00 ns
(-1.13%)Baseline: 36,478,776.12 ns
38,100,848.87 ns
(94.66%)
transfer/pacing-true/varying-seeds📈 view plot
🚷 view threshold		36,236,000.00 ns
(+1.04%)Baseline: 35,862,029.85 ns
37,479,328.18 ns
(96.68%)
🐰 View full continuous benchmarking report in Bencher

github-actions[bot] avatar Jun 20 '25 18:06 github-actions[bot]

🐰 Bencher Report

Branchbin-bench-refactor
Testbedt-linux64-ms-280
Click to view all benchmark results
BenchmarkLatencyBenchmark Result
milliseconds (ms)
(Result Δ%)		Upper Boundary
milliseconds (ms)
(Limit %)
s2n vs. neqo (cubic, paced)📈 view plot
🚷 view threshold		318.60 ms
(+1.47%)Baseline: 313.99 ms
330.07 ms
(96.52%)
🐰 View full continuous benchmarking report in Bencher

github-actions[bot] avatar Jun 20 '25 18:06 github-actions[bot]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 94.91%. Comparing base (27a42e4) to head (56125ed). Report is 3 commits behind head on main.

Additional details and impacted files 
@@            Coverage Diff             @@
##             main    #2599      +/-   ##
==========================================
+ Coverage   92.53%   94.91%   +2.38%     
==========================================
  Files         115      115              
  Lines       34014    34014              
  Branches    34014    34014              
==========================================
+ Hits        31474    32286     +812     
  Misses       1722     1722              
+ Partials      818        6     -812
Components Coverage Δ
neqo-common 97.45% <ø> (+1.00%) :arrow_up:
neqo-crypto 89.63% <ø> (+7.15%) :arrow_up:
neqo-http3 93.70% <ø> (+1.99%) :arrow_up:
neqo-qpack 95.45% <ø> (+2.06%) :arrow_up:
neqo-transport 95.99% <ø> (+2.13%) :arrow_up:
neqo-udp 90.86% <ø> (+11.16%) :arrow_up:

codecov[bot] avatar Jun 30 '25 09:06 codecov[bot]