neqo ci: Try `cargo-careful`

Now that https://github.com/RalfJung/cargo-careful/issues/35 got fixed.

May 02 '25 06:05 larseggert

Except that fix isn't in a release yet. v0.4.3 is too old.

May 02 '25 07:05 larseggert

Benchmark results

1-conn/1-100mb-resp/mtu-1504 (aka. Download)/client:

       time:   [694.67 ms 698.59 ms 702.50 ms]
       thrpt:  [142.35 MiB/s 143.15 MiB/s 143.95 MiB/s]

1-conn/10_000-parallel-1b-resp/mtu-1504 (aka. RPS)/client:

       time:   [351.58 ms 353.08 ms 354.59 ms]
       thrpt:  [28.202 Kelem/s 28.322 Kelem/s 28.443 Kelem/s]
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) low mild

1-conn/1-1b-resp/mtu-1504 (aka. HPS)/client:

       time:   [25.566 ms 25.735 ms 25.916 ms]
       thrpt:  [38.587  elem/s 38.858  elem/s 39.114  elem/s]
Found 4 outliers among 100 measurements (4.00%)
  4 (4.00%) high mild

1-conn/1-100mb-req/mtu-1504 (aka. Upload)/client:

       time:   [1.8442 s 1.8652 s 1.8874 s]
       thrpt:  [52.984 MiB/s 53.613 MiB/s 54.224 MiB/s]
Found 3 outliers among 100 measurements (3.00%)
  3 (3.00%) high mild

decode 4096 bytes, mask ff:

       time:   [12.090 µs 12.129 µs 12.175 µs]
Found 19 outliers among 100 measurements (19.00%)
  3 (3.00%) low severe
  3 (3.00%) low mild
  2 (2.00%) high mild
  11 (11.00%) high severe

decode 1048576 bytes, mask ff:

       time:   [3.1246 ms 3.1305 ms 3.1382 ms]
Found 5 outliers among 100 measurements (5.00%)
  1 (1.00%) low mild
  4 (4.00%) high severe

decode 4096 bytes, mask 7f:

       time:   [20.154 µs 20.205 µs 20.262 µs]
Found 20 outliers among 100 measurements (20.00%)
  1 (1.00%) low severe
  4 (4.00%) low mild
  2 (2.00%) high mild
  13 (13.00%) high severe

decode 1048576 bytes, mask 7f:

       time:   [5.2474 ms 5.2589 ms 5.2720 ms]
Found 14 outliers among 100 measurements (14.00%)
  14 (14.00%) high severe

decode 4096 bytes, mask 3f:

       time:   [7.0034 µs 7.0227 µs 7.0509 µs]
Found 14 outliers among 100 measurements (14.00%)
  4 (4.00%) low severe
  4 (4.00%) low mild
  2 (2.00%) high mild
  4 (4.00%) high severe

decode 1048576 bytes, mask 3f:

       time:   [1.7921 ms 1.7978 ms 1.8048 ms]
Found 8 outliers among 100 measurements (8.00%)
  1 (1.00%) low mild
  1 (1.00%) high mild
  6 (6.00%) high severe

1000 streams of 1 bytes/multistream:

       time:   [25.541 ms 25.566 ms 25.592 ms]
Found 52 outliers among 500 measurements (10.40%)
  51 (10.20%) high mild
  1 (0.20%) high severe

1000 streams of 1000 bytes/multistream:

       time:   [143.92 ms 143.95 ms 143.99 ms]
Found 3 outliers among 500 measurements (0.60%)
  3 (0.60%) high mild

coalesce_acked_from_zero 1+1 entries:

       time:   [94.504 ns 94.791 ns 95.075 ns]
Found 8 outliers among 100 measurements (8.00%)
  2 (2.00%) low mild
  6 (6.00%) high severe

coalesce_acked_from_zero 3+1 entries:

       time:   [112.41 ns 112.62 ns 112.85 ns]
Found 7 outliers among 100 measurements (7.00%)
  7 (7.00%) high severe

coalesce_acked_from_zero 10+1 entries:

       time:   [112.08 ns 112.50 ns 113.00 ns]
Found 25 outliers among 100 measurements (25.00%)
  5 (5.00%) low severe
  6 (6.00%) low mild
  4 (4.00%) high mild
  10 (10.00%) high severe

coalesce_acked_from_zero 1000+1 entries:

       time:   [92.563 ns 92.982 ns 93.433 ns]
Found 6 outliers among 100 measurements (6.00%)
  4 (4.00%) high mild
  2 (2.00%) high severe

RxStreamOrderer::inbound_frame():

       time:   [117.02 ms 117.07 ms 117.13 ms]
Found 16 outliers among 100 measurements (16.00%)
  2 (2.00%) low severe
  4 (4.00%) low mild
  10 (10.00%) high mild

SentPackets::take_ranges:

       time:   [8.3415 µs 8.5607 µs 8.7710 µs]
Found 19 outliers among 100 measurements (19.00%)
  3 (3.00%) low severe
  12 (12.00%) low mild
  3 (3.00%) high mild
  1 (1.00%) high severe

transfer/pacing-false/varying-seeds:

       time:   [35.714 ms 35.785 ms 35.853 ms]
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) low mild

transfer/pacing-true/varying-seeds:

       time:   [36.733 ms 36.822 ms 36.912 ms]

transfer/pacing-false/same-seed:

       time:   [35.648 ms 35.696 ms 35.744 ms]

transfer/pacing-true/same-seed:

       time:   [37.115 ms 37.174 ms 37.233 ms]

Client/server transfer results

Transfer of 33554432 bytes over loopback, 30 runs. All unit-less numbers are in milliseconds.

Client	Server	CC	Pacing	Mean ± σ	Min	Max	MiB/s ± σ	Δ `main`
neqo	neqo	reno	on	347.6 ± 93.1	298.3	776.1	:question:	:question:
neqo	neqo	reno		359.4 ± 110.2	291.8	831.0	:question:	:question:
neqo	neqo	cubic	on	347.2 ± 93.4	288.8	767.3	:question:	:question:
neqo	neqo	cubic		315.9 ± 13.7	298.1	361.0	:question:	:question:
google	neqo	reno	on	764.2 ± 90.7	566.0	956.1	:question:	:question:
google	neqo	reno		760.6 ± 86.7	564.4	895.5	:question:	:question:
google	neqo	cubic	on	765.7 ± 83.4	566.2	980.2	:question:	:question:
google	neqo	cubic		754.6 ± 84.8	564.6	856.9	:question:	:question:
google	google			571.2 ± 18.2	547.5	636.4	:question:	:question:
neqo	msquic	reno	on	271.6 ± 34.1	245.0	427.5	:question:	:question:
neqo	msquic	reno		267.0 ± 28.1	244.6	390.1	:question:	:question:
neqo	msquic	cubic	on	270.0 ± 36.3	243.1	448.8	:question:	:question:
neqo	msquic	cubic		259.1 ± 10.9	244.4	284.7	:question:	:question:
msquic	msquic			185.7 ± 31.4	162.7	336.8	:question:	:question:

:arrow_down: Download logs

May 02 '25 08:05 github-actions[bot]

Wasn't aware of this. Cool. In favor once fix is released.

May 05 '25 15:05 mxinden

Codecov Report

:white_check_mark: All modified and coverable lines are covered by tests. :white_check_mark: Project coverage is 93.43%. Comparing base (81c80ad) to head (5a9086d). :warning: Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2607      +/-   ##
==========================================
- Coverage   93.49%   93.43%   -0.06%     
==========================================
  Files         125      125              
  Lines       36637    36637              
  Branches    36637    36637              
==========================================
- Hits        34253    34232      -21     
- Misses       1532     1555      +23     
+ Partials      852      850       -2

Components	Coverage Δ
neqo-common	`97.40% <ø> (ø)`
neqo-crypto	`83.19% <ø> (-0.49%)`	:arrow_down:
neqo-http3	`93.26% <ø> (ø)`
neqo-qpack	`94.40% <ø> (ø)`
neqo-transport	`94.54% <ø> (-0.06%)`	:arrow_down:
neqo-udp	`78.94% <ø> (-0.48%)`	:arrow_down:
mtu	`85.44% <ø> (ø)`

Jul 03 '25 13:07 codecov[bot]

Blocked by https://github.com/RalfJung/cargo-careful/issues/39

Jul 03 '25 14:07 larseggert

Bencher Report

Branch	ci-cargo-careful
Testbed	On-prem

🚨 3 Alerts

Benchmark	Measure Units	View	Benchmark Result (Result Δ%)	Upper Boundary (Limit %)
coalesce_acked_from_zero 1000+1 entries	Latency nanoseconds (ns)	📈 plot 🚷 threshold 🚨 alert (🔔)	99.41 ns (+9.33%) Baseline: 90.93 ns	95.35 ns (104.25%)
decode 1048576 bytes, mask 3f	Latency milliseconds (ms)	📈 plot 🚷 threshold 🚨 alert (🔔)	3.19 ms (+82.09%) Baseline: 1.75 ms	2.48 ms (128.88%)
decode 4096 bytes, mask 3f	Latency microseconds (µs)	📈 plot 🚷 threshold 🚨 alert (🔔)	12.48 µs (+71.82%) Baseline: 7.26 µs	10.91 µs (114.35%)

Click to view all benchmark results

Benchmark	Latency	Benchmark Result nanoseconds (ns) (Result Δ%)	Upper Boundary nanoseconds (ns) (Limit %)
1-conn/1-100mb-req/mtu-1504 (aka. Upload)/client	📈 view plot 🚷 view threshold	209,910,000.00 ns (+1.02%) Baseline: 207,786,388.89 ns	216,943,809.70 ns (96.76%)
1-conn/1-100mb-resp/mtu-1504 (aka. Download)/client	📈 view plot 🚷 view threshold	202,080,000.00 ns (+0.03%) Baseline: 202,019,574.07 ns	211,724,147.69 ns (95.44%)
1-conn/1-1b-resp/mtu-1504 (aka. HPS)/client	📈 view plot 🚷 view threshold	38,900,000.00 ns (+13.77%) Baseline: 34,192,798.15 ns	46,131,123.79 ns (84.32%)
1-conn/10_000-parallel-1b-resp/mtu-1504 (aka. RPS)/client	📈 view plot 🚷 view threshold	284,510,000.00 ns (-1.52%) Baseline: 288,892,129.63 ns	302,135,638.47 ns (94.17%)
1-streams/each-1000-bytes/simulated-time	📈 view plot 🚷 view threshold	119,170,000.00 ns (+0.30%) Baseline: 118,817,796.30 ns	120,510,753.61 ns (98.89%)
1-streams/each-1000-bytes/wallclock-time	📈 view plot 🚷 view threshold	583,890.00 ns (-1.01%) Baseline: 589,842.57 ns	610,498.28 ns (95.64%)
1000-streams/each-1-bytes/simulated-time	📈 view plot 🚷 view threshold	2,330,800,000.00 ns (-75.37%) Baseline: 9,461,657,222.22 ns	24,117,317,262.39 ns (9.66%)
1000-streams/each-1-bytes/wallclock-time	📈 view plot 🚷 view threshold	12,475,000.00 ns (-6.57%) Baseline: 13,352,585.19 ns	15,176,151.37 ns (82.20%)
1000-streams/each-1000-bytes/simulated-time	📈 view plot 🚷 view threshold	16,569,000,000.00 ns (-7.06%) Baseline: 17,828,022,222.22 ns	20,850,292,932.82 ns (79.47%)
1000-streams/each-1000-bytes/wallclock-time	📈 view plot 🚷 view threshold	49,975,000.00 ns (-1.07%) Baseline: 50,514,705.56 ns	55,712,539.63 ns (89.70%)
RxStreamOrderer::inbound_frame()	📈 view plot 🚷 view threshold	109,900,000.00 ns (+0.15%) Baseline: 109,735,666.67 ns	111,381,933.30 ns (98.67%)
coalesce_acked_from_zero 1+1 entries	📈 view plot 🚷 view threshold	89.85 ns (+0.82%) Baseline: 89.11 ns	90.41 ns (99.37%)
coalesce_acked_from_zero 10+1 entries	📈 view plot 🚷 view threshold	105.61 ns (-0.40%) Baseline: 106.04 ns	107.11 ns (98.60%)
coalesce_acked_from_zero 1000+1 entries	📈 view plot 🚷 view threshold 🚨 view alert (🔔)	99.41 ns (+9.33%) Baseline: 90.93 ns	95.35 ns (104.25%)
coalesce_acked_from_zero 3+1 entries	📈 view plot 🚷 view threshold	106.60 ns (+0.06%) Baseline: 106.53 ns	107.53 ns (99.14%)
decode 1048576 bytes, mask 3f	📈 view plot 🚷 view threshold 🚨 view alert (🔔)	3,190,100.00 ns (+82.09%) Baseline: 1,751,891.85 ns	2,475,171.90 ns (128.88%)
decode 1048576 bytes, mask 7f	📈 view plot 🚷 view threshold	2,597,800.00 ns (-47.05%) Baseline: 4,906,236.85 ns	6,335,678.37 ns (41.00%)
decode 1048576 bytes, mask ff	📈 view plot 🚷 view threshold	1,844,200.00 ns (-37.44%) Baseline: 2,947,897.41 ns	3,629,669.94 ns (50.81%)
decode 4096 bytes, mask 3f	📈 view plot 🚷 view threshold 🚨 view alert (🔔)	12,480.00 ns (+71.82%) Baseline: 7,263.34 ns	10,914.13 ns (114.35%)
decode 4096 bytes, mask 7f	📈 view plot 🚷 view threshold	10,170.00 ns (-46.90%) Baseline: 19,151.74 ns	24,738.57 ns (41.11%)
decode 4096 bytes, mask ff	📈 view plot 🚷 view threshold	7,242.10 ns (-34.78%) Baseline: 11,103.87 ns	13,690.81 ns (52.90%)
sent::Packets::take_ranges	📈 view plot 🚷 view threshold	4,539.80 ns (-3.06%) Baseline: 4,683.25 ns	4,924.47 ns (92.19%)
transfer/pacing-false/same-seed/simulated-time/run	📈 view plot 🚷 view threshold	23,941,000,000.00 ns (-5.26%) Baseline: 25,269,429,368.03 ns	26,210,564,042.69 ns (91.34%)
transfer/pacing-false/same-seed/wallclock-time/run	📈 view plot 🚷 view threshold	23,932,000.00 ns (-5.44%) Baseline: 25,309,373.61 ns	27,092,768.85 ns (88.33%)
transfer/pacing-false/varying-seeds/simulated-time/run	📈 view plot 🚷 view threshold	23,941,000,000.00 ns (-4.65%) Baseline: 25,107,237,918.22 ns	25,793,410,948.82 ns (92.82%)
transfer/pacing-false/varying-seeds/wallclock-time/run	📈 view plot 🚷 view threshold	23,996,000.00 ns (-5.73%) Baseline: 25,453,280.67 ns	27,325,172.44 ns (87.82%)
transfer/pacing-true/same-seed/simulated-time/run	📈 view plot 🚷 view threshold	23,676,000,000.00 ns (-6.49%) Baseline: 25,320,468,401.49 ns	26,439,363,075.53 ns (89.55%)
transfer/pacing-true/same-seed/wallclock-time/run	📈 view plot 🚷 view threshold	24,092,000.00 ns (-8.98%) Baseline: 26,467,862.45 ns	28,855,235.61 ns (83.49%)
transfer/pacing-true/varying-seeds/simulated-time/run	📈 view plot 🚷 view threshold	23,676,000,000.00 ns (-4.98%) Baseline: 24,915,583,643.12 ns	25,645,589,302.56 ns (92.32%)
transfer/pacing-true/varying-seeds/wallclock-time/run	📈 view plot 🚷 view threshold	24,196,000.00 ns (-6.76%) Baseline: 25,950,724.91 ns	27,921,382.68 ns (86.66%)

🐰 View full continuous benchmarking report in Bencher

Dec 10 '25 12:12 github-actions[bot]

Failed Interop Tests

QUIC Interop Runner, client vs. server, differences relative to main at 81c80ad77811f6770606496f1d7799f52414f53d.

neqo-pr as client	neqo-pr as server
neqo-pr vs. aioquic: A neqo-pr vs. go-x-net: A BP BA neqo-pr vs. haproxy: A L1 BP BA neqo-pr vs. kwik: BP BA neqo-pr vs. linuxquic: A neqo-pr vs. lsquic: L1 C1 neqo-pr vs. msquic: A L1 C1 neqo-pr vs. mvfst: A L1 C1 :warning:BA neqo-pr vs. neqo: :warning:BA neqo-pr vs. nginx: A L1 C1 BP BA neqo-pr vs. ngtcp2: A :warning:L1 C1 CM neqo-pr vs. picoquic: A :rocket:C1 neqo-pr vs. quic-go: A :rocket:L1 :warning:C1 neqo-pr vs. quiche: A :warning:L1 BP BA neqo-pr vs. quinn: A :warning:L1 neqo-pr vs. s2n-quic: A L1 :warning:C1 BA CM neqo-pr vs. tquic: S A :warning:L1 BP BA neqo-pr vs. xquic: A :rocket:C1 :warning:L1	aioquic vs. neqo-pr: CM go-x-net vs. neqo-pr: CM kwik vs. neqo-pr: BP BA CM msquic vs. neqo-pr: CM mvfst vs. neqo-pr: Z A L1 C1 CM openssl vs. neqo-pr: LR M A CM quic-go vs. neqo-pr: CM quiche vs. neqo-pr: CM quinn vs. neqo-pr: V2 CM s2n-quic vs. neqo-pr: CM tquic vs. neqo-pr: CM xquic vs. neqo-pr: M CM

neqo-pr as client

neqo-pr as server

neqo-pr vs. aioquic: A
neqo-pr vs. go-x-net: A BP BA
neqo-pr vs. haproxy: A L1 BP BA
neqo-pr vs. kwik: BP BA
neqo-pr vs. linuxquic: A
neqo-pr vs. lsquic: L1 C1
neqo-pr vs. msquic: A L1 C1
neqo-pr vs. mvfst: A L1 C1 :warning:BA
neqo-pr vs. neqo: :warning:BA
neqo-pr vs. nginx: A L1 C1 BP BA
neqo-pr vs. ngtcp2: A :warning:L1 C1 CM
neqo-pr vs. picoquic: A :rocket:C1
neqo-pr vs. quic-go: A :rocket:L1 :warning:C1
neqo-pr vs. quiche: A :warning:L1 BP BA
neqo-pr vs. quinn: A :warning:L1
neqo-pr vs. s2n-quic: A L1 :warning:C1 BA CM
neqo-pr vs. tquic: S A :warning:L1 BP BA
neqo-pr vs. xquic: A :rocket:C1 :warning:L1

aioquic vs. neqo-pr: CM
go-x-net vs. neqo-pr: CM
kwik vs. neqo-pr: BP BA CM
msquic vs. neqo-pr: CM
mvfst vs. neqo-pr: Z A L1 C1 CM
openssl vs. neqo-pr: LR M A CM
quic-go vs. neqo-pr: CM
quiche vs. neqo-pr: CM
quinn vs. neqo-pr: V2 CM
s2n-quic vs. neqo-pr: CM
tquic vs. neqo-pr: CM
xquic vs. neqo-pr: M CM

All results

Succeeded Interop Tests

QUIC Interop Runner, client vs. server

neqo-pr as client

neqo-pr vs. aioquic: H DC LR C20 M S R Z 3 B U L1 L2 C1 C2 6 V2 BP BA
neqo-pr vs. go-x-net: H DC LR M B U L2 C2 6
neqo-pr vs. haproxy: H DC LR C20 M S R Z 3 B U L2 C1 C2 6 V2
neqo-pr vs. kwik: H DC LR C20 M S R Z 3 B U A L1 L2 C1 C2 6 V2
neqo-pr vs. linuxquic: H DC LR C20 M S R Z 3 B U E L1 L2 C1 C2 6 V2 BP BA CM
neqo-pr vs. lsquic: H DC LR C20 M S R Z 3 B U E A L2 C2 6 V2 BP BA CM
neqo-pr vs. msquic: H DC LR C20 M S R Z B U L2 C2 6 V2 BP BA
neqo-pr vs. mvfst: H DC LR M R Z 3 B U L2 C2 6 BP :warning:BA
neqo-pr vs. neqo: H DC LR C20 M S R Z 3 B U E A L1 L2 C1 C2 6 V2 BP :warning:BA CM
neqo-pr vs. nginx: H DC LR C20 M S R Z 3 B U L2 C2 6
neqo-pr vs. ngtcp2: H DC LR C20 M S R Z 3 B U E :warning:L1 L2 :warning:C1 C2 6 V2 BP BA
neqo-pr vs. picoquic: H DC LR C20 M S R Z 3 B U E L1 L2 :rocket:~~C1~~ C2 6 V2 BP BA
neqo-pr vs. quic-go: H DC LR C20 M S R Z 3 B U :rocket:~~L1~~ L2 :warning:C1 C2 6 BP BA
neqo-pr vs. quiche: H DC LR C20 M S R Z 3 B U :warning:L1 L2 C1 C2 6
neqo-pr vs. quinn: H DC LR C20 M S R Z 3 B U E :warning:L1 L2 C1 C2 6 BP BA
neqo-pr vs. s2n-quic: H DC LR C20 M S R 3 B U E L2 :warning:C1 C2 6 BP
neqo-pr vs. tquic: H DC LR C20 M R Z 3 B U :warning:L1 L2 C1 C2 6
neqo-pr vs. xquic: H DC LR C20 M S R Z 3 B U :warning:L1 L2 :rocket:~~C1~~ C2 6 BP BA

neqo-pr as server

aioquic vs. neqo-pr: H DC LR C20 M S R Z 3 B U A L1 L2 C1 C2 6 V2 BP BA
chrome vs. neqo-pr: 3
go-x-net vs. neqo-pr: H DC LR M B U A L2 C2 6 BP BA
kwik vs. neqo-pr: H DC LR C20 M S R Z 3 B U A L1 L2 C1 C2 6 V2
linuxquic vs. neqo-pr: H DC LR C20 M S R Z 3 B U E A L1 L2 C1 C2 6 V2 BP :rocket:~~BA~~ CM
lsquic vs. neqo-pr: H DC LR C20 M S R Z 3 B E A L1 L2 C1 C2 6 V2 BP BA CM
msquic vs. neqo-pr: H DC LR C20 M S R Z B U A L1 L2 C1 C2 6 V2 BP BA
mvfst vs. neqo-pr: H DC LR M 3 B L2 C2 6 BP BA
neqo vs. neqo-pr: H DC LR C20 M S R Z 3 B U E A L1 L2 C1 C2 6 V2 BP :rocket:~~BA~~ CM
ngtcp2 vs. neqo-pr: H DC LR C20 M S R Z 3 B U E A L1 L2 C1 C2 6 V2 BP BA CM
openssl vs. neqo-pr: H DC C20 S R 3 B L2 C2 6 BP BA
picoquic vs. neqo-pr: H DC LR C20 M S R Z 3 B U E A L1 L2 C1 C2 6 V2 BP BA CM
quic-go vs. neqo-pr: H DC LR C20 M S R Z 3 B U A L1 L2 C1 C2 6 BP BA
quiche vs. neqo-pr: H DC LR M S R Z 3 B A L1 L2 C1 C2 6 BP BA
quinn vs. neqo-pr: H DC LR C20 M S R Z 3 B U E A L1 L2 C1 C2 6 BP BA
s2n-quic vs. neqo-pr: H DC LR M S R 3 B E A L1 L2 C1 C2 6 BP BA
tquic vs. neqo-pr: H DC LR M S R Z 3 B A L1 L2 C1 C2 6 BP BA
xquic vs. neqo-pr: H DC LR C20 S R Z 3 B U A L1 L2 C1 C2 6 BP BA

Unsupported Interop Tests

QUIC Interop Runner, client vs. server

neqo-pr as client

neqo-pr vs. aioquic: E CM
neqo-pr vs. go-x-net: C20 S R Z 3 E L1 C1 V2 CM
neqo-pr vs. haproxy: E CM
neqo-pr vs. kwik: E CM
neqo-pr vs. msquic: 3 E CM
neqo-pr vs. mvfst: C20 S E V2 CM
neqo-pr vs. nginx: E V2 CM
neqo-pr vs. picoquic: CM
neqo-pr vs. quic-go: E V2 CM
neqo-pr vs. quiche: E V2 CM
neqo-pr vs. quinn: V2 CM
neqo-pr vs. s2n-quic: Z V2
neqo-pr vs. tquic: E V2 CM
neqo-pr vs. xquic: E V2 CM

neqo-pr as server

aioquic vs. neqo-pr: E
chrome vs. neqo-pr: H DC LR C20 M S R Z B U E A L1 L2 C1 C2 6 V2 BP BA CM
go-x-net vs. neqo-pr: C20 S R Z 3 E L1 C1 V2
kwik vs. neqo-pr: E
lsquic vs. neqo-pr: U
msquic vs. neqo-pr: 3 E
mvfst vs. neqo-pr: C20 S R U E V2
openssl vs. neqo-pr: Z U E L1 C1 V2
quic-go vs. neqo-pr: E V2
quiche vs. neqo-pr: C20 U E V2
s2n-quic vs. neqo-pr: C20 Z U V2
tquic vs. neqo-pr: C20 U E V2
xquic vs. neqo-pr: E V2

Dec 10 '25 13:12 github-actions[bot]

Benchmark results

Significant performance differences relative to 81c80ad77811f6770606496f1d7799f52414f53d.

1-conn/1-100mb-req/mtu-1504 (aka. Upload)/client: :broken_heart: Performance has regressed by +1.5894%.

       time:   [209.63 ms 209.91 ms 210.19 ms]
       thrpt:  [475.76 MiB/s 476.40 MiB/s 477.04 MiB/s]
change:
       time:   [+1.2409% +1.5894% +1.8873] (p = 0.00  -1.5646% -1.2257]
       Performance has regressed.
Found 2 outliers among 100 measurements (2.00%)
1 (1.00%) low mild
1 (1.00%) high severe

All results

1-conn/1-100mb-resp/mtu-1504 (aka. Download)/client: No change in performance detected.

       time:   [201.69 ms 202.08 ms 202.48 ms]
       thrpt:  [493.88 MiB/s 494.86 MiB/s 495.80 MiB/s]
change:
       time:   [-0.3731% -0.0470% +0.2387] (p = 0.76 > 0.05)
       thrpt:  [-0.2382% +0.0471% +0.3745]
       No change in performance detected.
Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) high mild

1-conn/10_000-parallel-1b-resp/mtu-1504 (aka. RPS)/client: Change within noise threshold.

       time:   [282.80 ms 284.51 ms 286.22 ms]
       thrpt:  [34.939 Kelem/s 35.148 Kelem/s 35.361 Kelem/s]
change:
       time:   [+0.4534% +1.3663% +2.3380] (p = 0.00  -1.3479% -0.4513]
       Change within noise threshold.

1-conn/1-1b-resp/mtu-1504 (aka. HPS)/client: No change in performance detected.

       time:   [38.722 ms 38.900 ms 39.095 ms]
       thrpt:  [25.579   B/s 25.707   B/s 25.825   B/s]
change:
       time:   [-0.7728% -0.0455% +0.7280] (p = 0.90 > 0.05)
       thrpt:  [-0.7227% +0.0455% +0.7788]
       No change in performance detected.
Found 11 outliers among 100 measurements (11.00%)
3 (3.00%) high mild
8 (8.00%) high severe

1-conn/1-100mb-req/mtu-1504 (aka. Upload)/client: :broken_heart: Performance has regressed by +1.5894%.

       time:   [209.63 ms 209.91 ms 210.19 ms]
       thrpt:  [475.76 MiB/s 476.40 MiB/s 477.04 MiB/s]
change:
       time:   [+1.2409% +1.5894% +1.8873] (p = 0.00  -1.5646% -1.2257]
       Performance has regressed.
Found 2 outliers among 100 measurements (2.00%)
1 (1.00%) low mild
1 (1.00%) high severe

decode 4096 bytes, mask ff: No change in performance detected.

       time:   [7.2314 µs 7.2421 µs 7.2529 µs]
       change: [-0.6100% -0.0425% +0.4210] (p = 0.89 > 0.05)
       No change in performance detected.
Found 4 outliers among 100 measurements (4.00%)
3 (3.00%) high mild
1 (1.00%) high severe

decode 1048576 bytes, mask ff: No change in performance detected.

       time:   [1.8415 ms 1.8442 ms 1.8471 ms]
       change: [-1.0206% -0.3263% +0.1266] (p = 0.34 > 0.05)
       No change in performance detected.
Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) high mild

decode 4096 bytes, mask 7f: No change in performance detected.

       time:   [10.157 µs 10.170 µs 10.184 µs]
       change: [-0.2866% +0.1190% +0.6746] (p = 0.68 > 0.05)
       No change in performance detected.
Found 7 outliers among 100 measurements (7.00%)
3 (3.00%) high mild
4 (4.00%) high severe

decode 1048576 bytes, mask 7f: No change in performance detected.

       time:   [2.5940 ms 2.5978 ms 2.6018 ms]
       change: [-0.2418% -0.0326% +0.1854] (p = 0.75 > 0.05)
       No change in performance detected.

decode 4096 bytes, mask 3f: No change in performance detected.

       time:   [12.460 µs 12.480 µs 12.504 µs]
       change: [-3.7741% -1.6910% -0.1100] (p = 0.06 > 0.05)
       No change in performance detected.
Found 6 outliers among 100 measurements (6.00%)
4 (4.00%) high mild
2 (2.00%) high severe

decode 1048576 bytes, mask 3f: Change within noise threshold.

       time:   [3.1852 ms 3.1901 ms 3.1951 ms]
       change: [-0.4687% -0.2362% -0.0082] (p = 0.05

1-streams/each-1000-bytes/wallclock-time: Change within noise threshold.

       time:   [581.91 µs 583.89 µs 586.98 µs]
       change: [-1.3359% -0.7464% -0.1256] (p = 0.02

1-streams/each-1000-bytes/simulated-time: No change in performance detected.

       time:   [118.96 ms 119.17 ms 119.38 ms]
       thrpt:  [8.1806 KiB/s 8.1949 KiB/s 8.2094 KiB/s]
change:
       time:   [-0.2499% -0.0036% +0.2378] (p = 0.98 > 0.05)
       thrpt:  [-0.2372% +0.0036% +0.2506]
       No change in performance detected.
Found 3 outliers among 100 measurements (3.00%)
3 (3.00%) low mild

1000-streams/each-1-bytes/wallclock-time: No change in performance detected.

       time:   [12.435 ms 12.475 ms 12.516 ms]
       change: [-0.7243% -0.2429% +0.2294] (p = 0.31 > 0.05)
       No change in performance detected.
Found 2 outliers among 100 measurements (2.00%)
2 (2.00%) high mild

1000-streams/each-1-bytes/simulated-time: No change in performance detected.

       time:   [2.3268 s 2.3308 s 2.3347 s]
       thrpt:  [428.32   B/s 429.04   B/s 429.77   B/s]
change:
       time:   [-0.2450% -0.0227% +0.2331] (p = 0.85 > 0.05)
       thrpt:  [-0.2325% +0.0227% +0.2456]
       No change in performance detected.

1000-streams/each-1000-bytes/wallclock-time: Change within noise threshold.

       time:   [49.869 ms 49.975 ms 50.081 ms]
       change: [-0.7442% -0.4298% -0.1333] (p = 0.01

1000-streams/each-1000-bytes/simulated-time: Change within noise threshold.

       time:   [16.319 s 16.569 s 16.823 s]
       thrpt:  [58.051 KiB/s 58.940 KiB/s 59.841 KiB/s]
change:
       time:   [+0.7205% +3.0300% +5.1979] (p = 0.01  -2.9409% -0.7154]
       Change within noise threshold.
Found 2 outliers among 100 measurements (2.00%)
1 (1.00%) low mild
1 (1.00%) high mild

coalesce_acked_from_zero 1+1 entries: No change in performance detected.

       time:   [89.074 ns 89.846 ns 91.071 ns]
       change: [-0.3454% +0.4547% +1.5120] (p = 0.38 > 0.05)
       No change in performance detected.
Found 11 outliers among 100 measurements (11.00%)
7 (7.00%) high mild
4 (4.00%) high severe

coalesce_acked_from_zero 3+1 entries: No change in performance detected.

       time:   [106.16 ns 106.60 ns 107.05 ns]
       change: [-1.0255% -0.0120% +0.7762] (p = 0.98 > 0.05)
       No change in performance detected.
Found 17 outliers among 100 measurements (17.00%)
1 (1.00%) high mild
16 (16.00%) high severe

coalesce_acked_from_zero 10+1 entries: No change in performance detected.

       time:   [105.23 ns 105.61 ns 106.09 ns]
       change: [-0.2737% +0.0248% +0.3079] (p = 0.88 > 0.05)
       No change in performance detected.
Found 9 outliers among 100 measurements (9.00%)
5 (5.00%) low mild
4 (4.00%) high severe

coalesce_acked_from_zero 1000+1 entries: No change in performance detected.

       time:   [91.353 ns 99.410 ns 118.65 ns]
       change: [-0.8261% +18.866% +76.213] (p = 0.58 > 0.05)
       No change in performance detected.
Found 15 outliers among 100 measurements (15.00%)
7 (7.00%) high mild
8 (8.00%) high severe

RxStreamOrderer::inbound_frame(): Change within noise threshold.

       time:   [109.82 ms 109.90 ms 109.98 ms]
       change: [-0.2266% -0.1340% -0.0494] (p = 0.01

sent::Packets::take_ranges: No change in performance detected.

       time:   [4.4530 µs 4.5398 µs 4.6125 µs]
       change: [-3.6444% -0.5876% +2.5313] (p = 0.71 > 0.05)
       No change in performance detected.
Found 2 outliers among 100 measurements (2.00%)
2 (2.00%) high mild

transfer/pacing-false/varying-seeds/wallclock-time/run: Change within noise threshold.

       time:   [23.978 ms 23.996 ms 24.015 ms]
       change: [+1.5710% +1.7982% +1.9613] (p = 0.00

transfer/pacing-false/varying-seeds/simulated-time/run: No change in performance detected.

       time:   [23.941 s 23.941 s 23.941 s]
       thrpt:  [171.09 KiB/s 171.09 KiB/s 171.09 KiB/s]
change:
       time:   [+0.0000% +0.0000% +0.0000] (p = NaN > 0.05)
       thrpt:  [+0.0000% +0.0000% +0.0000]
       No change in performance detected.

transfer/pacing-true/varying-seeds/wallclock-time/run: Change within noise threshold.

       time:   [24.169 ms 24.196 ms 24.227 ms]
       change: [+0.2356% +0.4429% +0.6202] (p = 0.00

transfer/pacing-true/varying-seeds/simulated-time/run: No change in performance detected.

       time:   [23.676 s 23.676 s 23.676 s]
       thrpt:  [173.01 KiB/s 173.01 KiB/s 173.01 KiB/s]
change:
       time:   [+0.0000% +0.0000% +0.0000] (p = NaN > 0.05)
       thrpt:  [+0.0000% +0.0000% +0.0000]
       No change in performance detected.

transfer/pacing-false/same-seed/wallclock-time/run: Change within noise threshold.

       time:   [23.908 ms 23.932 ms 23.962 ms]
       change: [+1.2380% +1.3873% +1.5378] (p = 0.00

transfer/pacing-false/same-seed/simulated-time/run: No change in performance detected.

       time:   [23.941 s 23.941 s 23.941 s]
       thrpt:  [171.09 KiB/s 171.09 KiB/s 171.09 KiB/s]
change:
       time:   [+0.0000% +0.0000% +0.0000] (p = NaN > 0.05)
       thrpt:  [+0.0000% +0.0000% +0.0000]
       No change in performance detected.

transfer/pacing-true/same-seed/wallclock-time/run: Change within noise threshold.

       time:   [24.058 ms 24.092 ms 24.142 ms]
       change: [-1.4552% -1.2145% -0.9605] (p = 0.00

transfer/pacing-true/same-seed/simulated-time/run: No change in performance detected.

       time:   [23.676 s 23.676 s 23.676 s]
       thrpt:  [173.01 KiB/s 173.01 KiB/s 173.01 KiB/s]
change:
       time:   [+0.0000% +0.0000% +0.0000] (p = NaN > 0.05)
       thrpt:  [+0.0000% +0.0000% +0.0000]
       No change in performance detected.

Download data for profiler.firefox.com or download performance comparison data.

Dec 10 '25 13:12 github-actions[bot]

Client/server transfer results

Performance differences relative to 81c80ad77811f6770606496f1d7799f52414f53d.

Transfer of 33554432 bytes over loopback, min. 100 runs. All unit-less numbers are in milliseconds.

Client vs. server (params)	Mean ± σ	Min	Max	MiB/s ± σ	Δ `main`	Δ `main`
neqo-neqo-cubic	95.9 ± 3.7	89.2	104.2	333.7 ± 8.6	:green_heart: -2.9	-3.0%
neqo-quiche-cubic	193.2 ± 4.2	187.1	203.8	165.6 ± 7.6	:broken_heart: 2.0	1.0%
s2n-neqo-cubic	173.3 ± 5.0	164.4	184.3	184.6 ± 6.4	:broken_heart: 1.6	0.9%

Table above only shows statistically significant changes. See all results below.

All results

Transfer of 33554432 bytes over loopback, min. 100 runs. All unit-less numbers are in milliseconds.

Client vs. server (params)	Mean ± σ	Min	Max	MiB/s ± σ	Δ `main`	Δ `main`
google-google-nopacing	460.1 ± 3.5	452.6	468.3	69.6 ± 9.1
google-neqo-cubic	271.7 ± 4.0	264.2	282.4	117.8 ± 8.0	0.1	0.0%
msquic-msquic-nopacing	192.0 ± 65.7	144.4	380.7	166.6 ± 0.5
msquic-neqo-cubic	228.2 ± 61.3	158.7	425.9	140.2 ± 0.5	8.0	3.6%
neqo-google-cubic	763.5 ± 7.7	754.6	825.8	41.9 ± 4.2	-0.2	-0.0%
neqo-msquic-cubic	159.5 ± 4.4	153.5	167.8	200.6 ± 7.3	-0.2	-0.1%
neqo-neqo-cubic	95.9 ± 3.7	89.2	104.2	333.7 ± 8.6	:green_heart: -2.9	-3.0%
neqo-neqo-cubic-nopacing	96.1 ± 3.8	88.3	105.1	332.8 ± 8.4	0.3	0.4%
neqo-neqo-reno	97.1 ± 4.5	87.8	108.4	329.5 ± 7.1	-1.1	-1.2%
neqo-neqo-reno-nopacing	96.8 ± 4.3	87.8	107.7	330.6 ± 7.4	-0.3	-0.3%
neqo-quiche-cubic	193.2 ± 4.2	187.1	203.8	165.6 ± 7.6	:broken_heart: 2.0	1.0%
neqo-s2n-cubic	220.3 ± 4.5	213.6	231.0	145.2 ± 7.1	-0.1	-0.0%
quiche-neqo-cubic	150.6 ± 5.4	140.5	172.1	212.6 ± 5.9	0.1	0.1%
quiche-quiche-nopacing	141.9 ± 4.0	135.4	153.1	225.4 ± 8.0
s2n-neqo-cubic	173.3 ± 5.0	164.4	184.3	184.6 ± 6.4	:broken_heart: 1.6	0.9%
s2n-s2n-nopacing	249.4 ± 26.3	229.1	368.2	128.3 ± 1.2

Download data for profiler.firefox.com or download performance comparison data.

Dec 10 '25 13:12 github-actions[bot]

Bencher Report

Branch	ci-cargo-careful
Testbed	On-prem

🚨 1 Alert

Iteration	Benchmark	Measure Units	View	Benchmark Result (Result Δ%)	Upper Boundary (Limit %)
3	neqo-msquic-cubic	Latency milliseconds (ms)	📈 plot 🚷 threshold 🚨 alert (🔔)	159.53 ms (+1.44%) Baseline: 157.26 ms	159.46 ms (100.04%)

Click to view all benchmark results

Benchmark	Latency	Benchmark Result milliseconds (ms) (Result Δ%)	Upper Boundary milliseconds (ms) (Limit %)
google-neqo-cubic	📈 view plot 🚷 view threshold	271.66 ms (-2.68%) Baseline: 279.15 ms	286.58 ms (94.80%)

Benchmark	Latency	Benchmark Result milliseconds (ms) (Result Δ%)	Upper Boundary milliseconds (ms) (Limit %)
msquic-neqo-cubic	📈 view plot 🚷 view threshold	228.23 ms (+9.06%) Baseline: 209.27 ms	242.59 ms (94.08%)

Benchmark	Latency	Benchmark Result milliseconds (ms) (Result Δ%)	Upper Boundary milliseconds (ms) (Limit %)
neqo-google-cubic	📈 view plot 🚷 view threshold	763.48 ms (-0.47%) Baseline: 767.06 ms	788.79 ms (96.79%)

Benchmark	Latency	Benchmark Result milliseconds (ms) (Result Δ%)	Upper Boundary milliseconds (ms) (Limit %)
neqo-msquic-cubic	📈 view plot 🚷 view threshold 🚨 view alert (🔔)	159.53 ms (+1.44%) Baseline: 157.26 ms	159.46 ms (100.04%)

Benchmark	Latency	Benchmark Result milliseconds (ms) (Result Δ%)	Upper Boundary milliseconds (ms) (Limit %)
neqo-neqo-cubic-nopacing	📈 view plot 🚷 view threshold	96.15 ms (-0.21%) Baseline: 96.35 ms	98.18 ms (97.93%)

Benchmark	Latency	Benchmark Result milliseconds (ms) (Result Δ%)	Upper Boundary milliseconds (ms) (Limit %)
neqo-neqo-cubic	📈 view plot 🚷 view threshold	95.88 ms (-1.62%) Baseline: 97.46 ms	99.82 ms (96.05%)

Benchmark	Latency	Benchmark Result milliseconds (ms) (Result Δ%)	Upper Boundary milliseconds (ms) (Limit %)
neqo-neqo-reno-nopacing	📈 view plot 🚷 view threshold	96.78 ms (+0.07%) Baseline: 96.72 ms	99.20 ms (97.56%)

Benchmark	Latency	Benchmark Result milliseconds (ms) (Result Δ%)	Upper Boundary milliseconds (ms) (Limit %)
neqo-neqo-reno	📈 view plot 🚷 view threshold	97.13 ms (-0.44%) Baseline: 97.55 ms	99.67 ms (97.45%)

Benchmark	Latency	Benchmark Result milliseconds (ms) (Result Δ%)	Upper Boundary milliseconds (ms) (Limit %)
neqo-quiche-cubic	📈 view plot 🚷 view threshold	193.24 ms (+0.44%) Baseline: 192.40 ms	195.12 ms (99.04%)

Benchmark	Latency	Benchmark Result milliseconds (ms) (Result Δ%)	Upper Boundary milliseconds (ms) (Limit %)
neqo-s2n-cubic	📈 view plot 🚷 view threshold	220.35 ms (-0.13%) Baseline: 220.63 ms	224.10 ms (98.33%)

Benchmark	Latency	Benchmark Result milliseconds (ms) (Result Δ%)	Upper Boundary milliseconds (ms) (Limit %)
quiche-neqo-cubic	📈 view plot 🚷 view threshold	150.55 ms (-2.03%) Baseline: 153.67 ms	156.75 ms (96.05%)

Benchmark	Latency	Benchmark Result milliseconds (ms) (Result Δ%)	Upper Boundary milliseconds (ms) (Limit %)
s2n-neqo-cubic	📈 view plot 🚷 view threshold	173.31 ms (-0.26%) Baseline: 173.77 ms	177.36 ms (97.72%)

🐰 View full continuous benchmarking report in Bencher

Dec 10 '25 13:12 github-actions[bot]