neqo ci: add windows-11-arm runner

GitHub Actions now provide Windows ARM64 runners:

https://github.blog/changelog/2025-04-14-windows-arm64-hosted-runners-now-available-in-public-preview/

Related: https://bugzilla.mozilla.org/show_bug.cgi?id=1931918

Apr 17 '25 09:04 mxinden

rustup not being there may be due to https://github.com/actions/partner-runner-images/issues/77

Maybe we just install it?

Apr 17 '25 10:04 larseggert

Benchmark results

Performance differences relative to 847a5989f20d30eb3675d1435180fd48a8c6a3ad.

1-conn/1-100mb-resp/mtu-1504 (aka. Download)/client: No change in performance detected.

       time:   [693.78 ms 697.61 ms 701.42 ms]
       thrpt:  [142.57 MiB/s 143.35 MiB/s 144.14 MiB/s]
change:
       time:   [-1.5580% -0.7959% +0.0525%] (p = 0.05 > 0.05)
       thrpt:  [-0.0524% +0.8022% +1.5826%]

1-conn/10_000-parallel-1b-resp/mtu-1504 (aka. RPS)/client: No change in performance detected.

       time:   [352.25 ms 353.62 ms 354.98 ms]
       thrpt:  [28.171 Kelem/s 28.279 Kelem/s 28.389 Kelem/s]
change:
       time:   [-0.3659% +0.2205% +0.7982%] (p = 0.47 > 0.05)
       thrpt:  [-0.7918% -0.2200% +0.3673%]

1-conn/1-1b-resp/mtu-1504 (aka. HPS)/client: No change in performance detected.

       time:   [25.488 ms 25.633 ms 25.785 ms]
       thrpt:  [38.782  elem/s 39.012  elem/s 39.234  elem/s]
change:
       time:   [-1.3696% -0.5205% +0.2719%] (p = 0.22 > 0.05)
       thrpt:  [-0.2712% +0.5232% +1.3886%]
Found 4 outliers among 100 measurements (4.00%)
4 (4.00%) high mild

1-conn/1-100mb-req/mtu-1504 (aka. Upload)/client: :green_heart: Performance has improved.

       time:   [1.7915 s 1.8133 s 1.8361 s]
       thrpt:  [54.463 MiB/s 55.147 MiB/s 55.820 MiB/s]
change:
       time:   [-8.3956% -6.6210% -4.8312%] (p = 0.00 +7.0904% +9.1650%]
Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) high mild

decode 4096 bytes, mask ff: No change in performance detected.

       time:   [12.070 µs 12.100 µs 12.138 µs]
       change: [-0.4081% -0.1094% +0.1954%] (p = 0.49 > 0.05)
Found 12 outliers among 100 measurements (12.00%)
3 (3.00%) low severe
2 (2.00%) low mild
1 (1.00%) high mild
6 (6.00%) high severe

decode 1048576 bytes, mask ff: No change in performance detected.

       time:   [3.1272 ms 3.1348 ms 3.1442 ms]
       change: [-0.7339% -0.2697% +0.1582%] (p = 0.26 > 0.05)
Found 6 outliers among 100 measurements (6.00%)
6 (6.00%) high severe

decode 4096 bytes, mask 7f: No change in performance detected.

       time:   [20.168 µs 20.224 µs 20.286 µs]
       change: [-0.3661% +0.0032% +0.4244%] (p = 0.98 > 0.05)
Found 21 outliers among 100 measurements (21.00%)
1 (1.00%) low severe
4 (4.00%) low mild
2 (2.00%) high mild
14 (14.00%) high severe

decode 1048576 bytes, mask 7f: No change in performance detected.

       time:   [5.2471 ms 5.2602 ms 5.2751 ms]
       change: [-0.3443% -0.0005% +0.3593%] (p = 1.00 > 0.05)
Found 14 outliers among 100 measurements (14.00%)
1 (1.00%) low mild
13 (13.00%) high severe

decode 4096 bytes, mask 3f: No change in performance detected.

       time:   [7.0074 µs 7.0185 µs 7.0385 µs]
       change: [-0.8880% -0.1196% +0.7775%] (p = 0.80 > 0.05)
Found 15 outliers among 100 measurements (15.00%)
5 (5.00%) low severe
2 (2.00%) low mild
1 (1.00%) high mild
7 (7.00%) high severe

decode 1048576 bytes, mask 3f: No change in performance detected.

       time:   [1.7922 ms 1.7980 ms 1.8051 ms]
       change: [-0.6316% -0.0830% +0.4601%] (p = 0.75 > 0.05)
Found 7 outliers among 100 measurements (7.00%)
1 (1.00%) high mild
6 (6.00%) high severe

1000 streams of 1 bytes/multistream: No change in performance detected.

       time:   [25.576 ms 25.602 ms 25.628 ms]
       change: [-0.0568% +0.0869% +0.2311%] (p = 0.23 > 0.05)
Found 57 outliers among 500 measurements (11.40%)
53 (10.60%) high mild
4 (0.80%) high severe

1000 streams of 1000 bytes/multistream: Change within noise threshold.

       time:   [143.92 ms 143.96 ms 143.99 ms]
       change: [+0.0182% +0.0530% +0.0891%] (p = 0.00 Found 2 outliers among 500 measurements (0.40%)
2 (0.40%) high mild

coalesce_acked_from_zero 1+1 entries: No change in performance detected.

       time:   [94.809 ns 95.147 ns 95.482 ns]
       change: [-0.4014% +0.0250% +0.4502%] (p = 0.91 > 0.05)
Found 13 outliers among 100 measurements (13.00%)
13 (13.00%) high mild

coalesce_acked_from_zero 3+1 entries: No change in performance detected.

       time:   [112.69 ns 113.02 ns 113.37 ns]
       change: [-0.6434% -0.1758% +0.2601%] (p = 0.46 > 0.05)
Found 16 outliers among 100 measurements (16.00%)
1 (1.00%) low severe
2 (2.00%) low mild
13 (13.00%) high severe

coalesce_acked_from_zero 10+1 entries: No change in performance detected.

       time:   [111.94 ns 112.29 ns 112.74 ns]
       change: [-0.5043% +0.2238% +0.9630%] (p = 0.59 > 0.05)
Found 15 outliers among 100 measurements (15.00%)
4 (4.00%) low severe
4 (4.00%) low mild
7 (7.00%) high severe

coalesce_acked_from_zero 1000+1 entries: No change in performance detected.

       time:   [92.961 ns 93.453 ns 93.987 ns]
       change: [-0.5859% +0.4267% +1.4132%] (p = 0.41 > 0.05)
Found 6 outliers among 100 measurements (6.00%)
5 (5.00%) high mild
1 (1.00%) high severe

RxStreamOrderer::inbound_frame(): Change within noise threshold.

       time:   [117.33 ms 117.38 ms 117.44 ms]
       change: [+0.3764% +0.4410% +0.5103%] (p = 0.00 Found 15 outliers among 100 measurements (15.00%)
1 (1.00%) low severe
5 (5.00%) low mild
9 (9.00%) high mild

SentPackets::take_ranges: No change in performance detected.

       time:   [8.2627 µs 8.5307 µs 8.7854 µs]
       change: [-3.2255% -0.6526% +2.0184%] (p = 0.63 > 0.05)
Found 23 outliers among 100 measurements (23.00%)
1 (1.00%) low severe
18 (18.00%) low mild
4 (4.00%) high mild

transfer/pacing-false/varying-seeds: Change within noise threshold.

       time:   [35.824 ms 35.885 ms 35.945 ms]
       change: [+0.7514% +1.0002% +1.2374%] (p = 0.00

transfer/pacing-true/varying-seeds: Change within noise threshold.

       time:   [36.695 ms 36.799 ms 36.903 ms]
       change: [+0.1973% +0.6341% +1.0569%] (p = 0.00

transfer/pacing-false/same-seed: Change within noise threshold.

       time:   [35.490 ms 35.542 ms 35.597 ms]
       change: [+0.5478% +0.7464% +0.9507%] (p = 0.00 Found 4 outliers among 100 measurements (4.00%)
4 (4.00%) high mild

transfer/pacing-true/same-seed: Change within noise threshold.

       time:   [37.377 ms 37.444 ms 37.512 ms]
       change: [+0.2418% +0.4769% +0.7140%] (p = 0.00 Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) low mild

Client/server transfer results

Performance differences relative to 847a5989f20d30eb3675d1435180fd48a8c6a3ad.

Transfer of 33554432 bytes over loopback, 30 runs. All unit-less numbers are in milliseconds.

Client	Server	CC	Pacing	Mean ± σ	Min	Max	MiB/s ± σ	Δ `main`	Δ `main`
neqo	neqo	reno	on	329.9 ± 46.9	287.4	514.3	97.0 ± 0.7	-3.4	-1.0%
neqo	neqo	reno		345.5 ± 111.8	287.9	848.4	92.6 ± 0.3	-10.0	-2.8%
neqo	neqo	cubic	on	330.1 ± 48.8	299.6	508.4	96.9 ± 0.7	8.0	2.5%
neqo	neqo	cubic		333.7 ± 51.3	294.7	498.3	95.9 ± 0.6	3.2	1.0%
google	neqo	reno	on	769.7 ± 88.4	571.2	918.9	41.6 ± 0.4	8.0	1.0%
google	neqo	reno		771.5 ± 90.3	547.4	946.8	41.5 ± 0.4	11.7	1.5%
google	neqo	cubic	on	775.1 ± 109.7	575.2	1185.1	41.3 ± 0.3	19.7	2.6%
google	neqo	cubic		757.9 ± 83.4	556.9	886.6	42.2 ± 0.4	-4.3	-0.6%
google	google			575.5 ± 41.0	551.8	781.3	55.6 ± 0.8	2.7	0.5%
neqo	msquic	reno	on	273.6 ± 42.3	243.9	430.1	117.0 ± 0.8	4.9	1.8%
neqo	msquic	reno		263.9 ± 18.6	243.4	312.4	121.2 ± 1.7	-9.9	-3.6%
neqo	msquic	cubic	on	270.9 ± 36.5	245.6	434.1	118.1 ± 0.9	7.5	2.9%
neqo	msquic	cubic		267.9 ± 38.0	240.8	448.0	119.5 ± 0.8	0.5	0.2%
msquic	msquic			196.8 ± 39.3	159.3	339.8	162.6 ± 0.8	0.1	0.1%

:arrow_down: Download logs

Apr 17 '25 10:04 github-actions[bot]

This is now stuck on https://github.com/actions/partner-runner-images/issues/90

Apr 24 '25 09:04 larseggert

Codecov Report

:white_check_mark: All modified and coverable lines are covered by tests. :white_check_mark: Project coverage is 93.37%. Comparing base (4c741fe) to head (616621d).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2591      +/-   ##
==========================================
- Coverage   93.41%   93.37%   -0.04%     
==========================================
  Files         123      123              
  Lines       35706    35706              
  Branches    35706    35706              
==========================================
- Hits        33353    33341      -12     
- Misses       1507     1519      +12     
  Partials      846      846

Components	Coverage Δ
neqo-common	`97.30% <ø> (ø)`
neqo-crypto	`83.22% <ø> (-0.49%)`	:arrow_down:
neqo-http3	`93.32% <ø> (ø)`
neqo-qpack	`94.15% <ø> (ø)`
neqo-transport	`94.48% <ø> (-0.02%)`	:arrow_down:
neqo-udp	`80.48% <ø> (+0.48%)`	:arrow_up:
mtu	`85.57% <ø> (-0.20%)`	:arrow_down:

Jul 19 '25 10:07 codecov[bot]

There is a new version of the Win11 ARM image, but it's still shipping with the amd64 llvm toolchain.

Jul 19 '25 10:07 larseggert

Now blocked because NSPR/NSS don't build on Windows/ARM. Sigh.

Aug 27 '25 09:08 larseggert

Bencher Report

Branch	ci-windows-arm
Testbed	On-prem

Click to view all benchmark results

Benchmark	Latency	Benchmark Result milliseconds (ms) (Result Δ%)	Upper Boundary milliseconds (ms) (Limit %)
google vs. neqo (cubic, paced)	📈 view plot 🚷 view threshold	277.72 ms (+0.01%) Baseline: 277.70 ms	280.28 ms (99.09%)

Benchmark	Latency	Benchmark Result milliseconds (ms) (Result Δ%)	Upper Boundary milliseconds (ms) (Limit %)
msquic vs. neqo (cubic, paced)	📈 view plot 🚷 view threshold	191.20 ms (-2.40%) Baseline: 195.91 ms	230.61 ms (82.91%)

Benchmark	Latency	Benchmark Result milliseconds (ms) (Result Δ%)	Upper Boundary milliseconds (ms) (Limit %)
neqo vs. google (cubic, paced)	📈 view plot 🚷 view threshold	758.90 ms (+0.12%) Baseline: 757.98 ms	764.84 ms (99.22%)

Benchmark	Latency	Benchmark Result milliseconds (ms) (Result Δ%)	Upper Boundary milliseconds (ms) (Limit %)
neqo vs. msquic (cubic, paced)	📈 view plot 🚷 view threshold	158.24 ms (+0.89%) Baseline: 156.84 ms	158.80 ms (99.65%)

Benchmark	Latency	Benchmark Result milliseconds (ms) (Result Δ%)	Upper Boundary milliseconds (ms) (Limit %)
neqo vs. neqo (cubic)	📈 view plot 🚷 view threshold	88.43 ms (-3.13%) Baseline: 91.29 ms	94.97 ms (93.12%)

Benchmark	Latency	Benchmark Result milliseconds (ms) (Result Δ%)	Upper Boundary milliseconds (ms) (Limit %)
neqo vs. neqo (cubic, paced)	📈 view plot 🚷 view threshold	90.12 ms (-2.70%) Baseline: 92.63 ms	95.84 ms (94.04%)

Benchmark	Latency	Benchmark Result milliseconds (ms) (Result Δ%)	Upper Boundary milliseconds (ms) (Limit %)
neqo vs. neqo (reno)	📈 view plot 🚷 view threshold	88.81 ms (-2.60%) Baseline: 91.18 ms	94.43 ms (94.05%)

Benchmark	Latency	Benchmark Result milliseconds (ms) (Result Δ%)	Upper Boundary milliseconds (ms) (Limit %)
neqo vs. neqo (reno, paced)	📈 view plot 🚷 view threshold	92.17 ms (-0.41%) Baseline: 92.55 ms	95.62 ms (96.40%)

Benchmark	Latency	Benchmark Result milliseconds (ms) (Result Δ%)	Upper Boundary milliseconds (ms) (Limit %)
neqo vs. quiche (cubic, paced)	📈 view plot 🚷 view threshold	193.42 ms (-0.22%) Baseline: 193.84 ms	197.20 ms (98.08%)

Benchmark	Latency	Benchmark Result milliseconds (ms) (Result Δ%)	Upper Boundary milliseconds (ms) (Limit %)
neqo vs. s2n (cubic, paced)	📈 view plot 🚷 view threshold	218.51 ms (-1.05%) Baseline: 220.82 ms	223.54 ms (97.75%)

Benchmark	Latency	Benchmark Result milliseconds (ms) (Result Δ%)	Upper Boundary milliseconds (ms) (Limit %)
quiche vs. neqo (cubic, paced)	📈 view plot 🚷 view threshold	153.72 ms (+1.36%) Baseline: 151.67 ms	157.35 ms (97.70%)

Benchmark	Latency	Benchmark Result milliseconds (ms) (Result Δ%)	Upper Boundary milliseconds (ms) (Limit %)
s2n vs. neqo (cubic, paced)	📈 view plot 🚷 view threshold	176.49 ms (+1.76%) Baseline: 173.44 ms	177.40 ms (99.49%)

🐰 View full continuous benchmarking report in Bencher

Sep 19 '25 05:09 github-actions[bot]

Bencher Report

Branch	ci-windows-arm
Testbed	On-prem

Click to view all benchmark results

Benchmark	Latency	Benchmark Result nanoseconds (ns) (Result Δ%)	Upper Boundary nanoseconds (ns) (Limit %)
1-conn/1-100mb-req/mtu-1504 (aka. Upload)/client	📈 view plot 🚷 view threshold	205,880,000.00 ns (-2.28%) Baseline: 210,690,384.62 ns	218,437,963.37 ns (94.25%)
1-conn/1-100mb-resp/mtu-1504 (aka. Download)/client	📈 view plot 🚷 view threshold	201,090,000.00 ns (-1.93%) Baseline: 205,050,000.00 ns	213,678,455.81 ns (94.11%)
1-conn/1-1b-resp/mtu-1504 (aka. HPS)/client	📈 view plot 🚷 view threshold	28,519,000.00 ns (+0.62%) Baseline: 28,342,826.92 ns	28,865,643.95 ns (98.80%)
1-conn/10_000-parallel-1b-resp/mtu-1504 (aka. RPS)/client	📈 view plot 🚷 view threshold	296,740,000.00 ns (+0.77%) Baseline: 294,484,807.69 ns	302,046,132.74 ns (98.24%)
1-streams/each-1000-bytes/simulated-time	📈 view plot 🚷 view threshold	118,940,000.00 ns (+0.97%) Baseline: 117,792,019.23 ns	120,565,831.25 ns (98.65%)
1-streams/each-1000-bytes/wallclock-time	📈 view plot 🚷 view threshold	589,100.00 ns (-2.39%) Baseline: 603,503.27 ns	628,297.90 ns (93.76%)
1000-streams/each-1-bytes/simulated-time	📈 view plot 🚷 view threshold	15,004,000,000.00 ns (+0.09%) Baseline: 14,990,278,846.15 ns	15,008,780,143.22 ns (99.97%)
1000-streams/each-1-bytes/wallclock-time	📈 view plot 🚷 view threshold	14,060,000.00 ns (-2.40%) Baseline: 14,406,288.46 ns	15,025,356.49 ns (93.58%)
1000-streams/each-1000-bytes/simulated-time	📈 view plot 🚷 view threshold	18,989,000,000.00 ns (+0.50%) Baseline: 18,894,288,461.54 ns	19,104,503,469.82 ns (99.40%)
1000-streams/each-1000-bytes/wallclock-time	📈 view plot 🚷 view threshold	50,690,000.00 ns (-5.75%) Baseline: 53,783,615.38 ns	60,041,794.13 ns (84.42%)
RxStreamOrderer::inbound_frame()	📈 view plot 🚷 view threshold	110,210,000.00 ns (+0.43%) Baseline: 109,735,480.77 ns	111,783,063.88 ns (98.59%)
coalesce_acked_from_zero 1+1 entries	📈 view plot 🚷 view threshold	88.41 ns (-0.24%) Baseline: 88.62 ns	89.30 ns (98.99%)
coalesce_acked_from_zero 10+1 entries	📈 view plot 🚷 view threshold	105.97 ns (-0.16%) Baseline: 106.14 ns	107.06 ns (98.98%)
coalesce_acked_from_zero 1000+1 entries	📈 view plot 🚷 view threshold	88.97 ns (-0.88%) Baseline: 89.77 ns	93.40 ns (95.26%)
coalesce_acked_from_zero 3+1 entries	📈 view plot 🚷 view threshold	106.33 ns (-0.30%) Baseline: 106.65 ns	107.59 ns (98.83%)
decode 1048576 bytes, mask 3f	📈 view plot 🚷 view threshold	1,593,900.00 ns (+0.08%) Baseline: 1,592,592.31 ns	1,599,861.60 ns (99.63%)
decode 1048576 bytes, mask 7f	📈 view plot 🚷 view threshold	5,042,100.00 ns (-0.32%) Baseline: 5,058,516.35 ns	5,078,167.99 ns (99.29%)
decode 1048576 bytes, mask ff	📈 view plot 🚷 view threshold	3,033,700.00 ns (+0.04%) Baseline: 3,032,604.81 ns	3,044,910.84 ns (99.63%)
decode 4096 bytes, mask 3f	📈 view plot 🚷 view threshold	8,334.20 ns (+0.49%) Baseline: 8,293.43 ns	8,334.23 ns (100.00%)
decode 4096 bytes, mask 7f	📈 view plot 🚷 view threshold	20,002.00 ns (-0.01%) Baseline: 20,004.38 ns	20,085.96 ns (99.58%)
decode 4096 bytes, mask ff	📈 view plot 🚷 view threshold	11,635.00 ns (-1.34%) Baseline: 11,792.71 ns	12,019.95 ns (96.80%)
sent::Packets::take_ranges	📈 view plot 🚷 view threshold	4,515.80 ns (-5.55%) Baseline: 4,781.40 ns	5,032.18 ns (89.74%)
transfer/pacing-false/same-seed/simulated-time/run	📈 view plot 🚷 view threshold	25,152,000,000.00 ns
transfer/pacing-false/same-seed/wallclock-time/run	📈 view plot 🚷 view threshold	26,260,000.00 ns (+0.68%) Baseline: 26,082,784.31 ns	26,917,351.31 ns (97.56%)
transfer/pacing-false/varying-seeds/simulated-time/run	📈 view plot 🚷 view threshold	25,189,000,000.00 ns (+0.10%) Baseline: 25,164,578,431.37 ns	25,208,456,587.39 ns (99.92%)
transfer/pacing-false/varying-seeds/wallclock-time/run	📈 view plot 🚷 view threshold	26,807,000.00 ns (+1.38%) Baseline: 26,441,754.90 ns	27,241,844.10 ns (98.40%)
transfer/pacing-true/same-seed/simulated-time/run	📈 view plot 🚷 view threshold	25,588,000,000.00 ns
transfer/pacing-true/same-seed/wallclock-time/run	📈 view plot 🚷 view threshold	27,902,000.00 ns (+1.12%) Baseline: 27,592,598.04 ns	28,431,780.31 ns (98.14%)
transfer/pacing-true/varying-seeds/simulated-time/run	📈 view plot 🚷 view threshold	24,982,000,000.00 ns (-0.05%) Baseline: 24,995,509,803.92 ns	25,047,679,065.39 ns (99.74%)
transfer/pacing-true/varying-seeds/wallclock-time/run	📈 view plot 🚷 view threshold	27,313,000.00 ns (+1.21%) Baseline: 26,986,813.73 ns	27,767,409.42 ns (98.36%)

🐰 View full continuous benchmarking report in Bencher

Sep 19 '25 05:09 github-actions[bot]

Failed Interop Tests

QUIC Interop Runner, client vs. server, differences relative to 836046b230c4052e3f4287e0a068ae0a13fd68b2.

neqo-latest as client

neqo-latest vs. go-x-net: BP BA
neqo-latest vs. haproxy: :warning:C1 BP BA
neqo-latest vs. kwik: BP BA
neqo-latest vs. linuxquic: L1 C1
neqo-latest vs. lsquic: run cancelled after 20 min
neqo-latest vs. msquic: :rocket:~~R~~ Z A L1 :warning:L2 C1
neqo-latest vs. mvfst: A L1 C1 :rocket:~~BA~~
neqo-latest vs. neqo: A
neqo-latest vs. neqo-latest: A
neqo-latest vs. nginx: BP BA
neqo-latest vs. ngtcp2: E CM
neqo-latest vs. picoquic: :warning:Z E A L1 C1
neqo-latest vs. quic-go: A
neqo-latest vs. quiche: BP BA
neqo-latest vs. quinn: A
neqo-latest vs. s2n-quic: E BA CM
neqo-latest vs. tquic: S BP BA
neqo-latest vs. xquic: :warning:A L1 C1

neqo-latest as server

aioquic vs. neqo-latest: CM
go-x-net vs. neqo-latest: CM
kwik vs. neqo-latest: BP BA CM
msquic vs. neqo-latest: :rocket:~~U BP~~ CM
mvfst vs. neqo-latest: Z A L1 C1 CM
neqo vs. neqo-latest: A
openssl vs. neqo-latest: LR M A CM
quic-go vs. neqo-latest: CM
quiche vs. neqo-latest: CM
quinn vs. neqo-latest: V2 CM
s2n-quic vs. neqo-latest: CM
tquic vs. neqo-latest: CM
xquic vs. neqo-latest: M CM

All results

Succeeded Interop Tests

QUIC Interop Runner, client vs. server

neqo-latest as client

neqo-latest vs. aioquic: H DC LR C20 M S R Z 3 B U A L1 L2 C1 C2 6 V2 BP BA
neqo-latest vs. go-x-net: H DC LR M B U A L2 C2 6
neqo-latest vs. haproxy: H DC LR C20 M S R Z 3 B U A L1 L2 :warning:C1 C2 6 V2
neqo-latest vs. kwik: H DC LR C20 M S R Z 3 B U A L1 L2 C1 C2 6 V2
neqo-latest vs. linuxquic: H DC LR C20 M S R Z 3 B U E A L2 C2 6 V2 BP BA CM
neqo-latest vs. msquic: H DC LR C20 M S :rocket:~~R~~ B U :warning:L2 C2 6 V2 BP BA
neqo-latest vs. mvfst: H DC LR M R Z 3 B U L2 C2 6 BP :rocket:~~BA~~
neqo-latest vs. neqo: H DC LR C20 M S R Z 3 B U E L1 L2 C1 C2 6 V2 BP BA CM
neqo-latest vs. neqo-latest: H DC LR C20 M S R Z 3 B U E L1 L2 C1 C2 6 V2 BP BA CM
neqo-latest vs. nginx: H DC LR C20 M S R Z 3 B U A L1 L2 C1 C2 6
neqo-latest vs. ngtcp2: H DC LR C20 M S R Z 3 B U A L1 L2 C1 C2 6 V2 BP BA
neqo-latest vs. picoquic: H DC LR C20 M S R :warning:Z 3 B U L2 C2 6 V2 BP BA
neqo-latest vs. quic-go: H DC LR C20 M S R Z 3 B U L1 L2 C1 C2 6 BP BA
neqo-latest vs. quiche: H DC LR C20 M S R Z 3 B U A L1 L2 C1 C2 6
neqo-latest vs. quinn: H DC LR C20 M S R Z 3 B U E L1 L2 C1 C2 6 BP BA
neqo-latest vs. s2n-quic: H DC LR C20 M S R 3 B U A L1 L2 C1 C2 6 BP
neqo-latest vs. tquic: H DC LR C20 M R Z 3 B U A L1 L2 C1 C2 6
neqo-latest vs. xquic: :rocket:~~H DC LR C20 M R Z 3 B U L2 C2 6 BP BA~~

neqo-latest as server

aioquic vs. neqo-latest: H DC LR C20 M S R Z 3 B A L1 L2 C1 C2 6 V2 BP BA
chrome vs. neqo-latest: 3
go-x-net vs. neqo-latest: H DC LR M B U A L2 C2 6 BP BA
kwik vs. neqo-latest: H DC LR C20 M S R Z 3 B U A L1 L2 C1 C2 6 V2
linuxquic vs. neqo-latest: H DC LR C20 M S R Z 3 B U E A L1 L2 C1 C2 6 V2 BP BA CM
lsquic vs. neqo-latest: H DC LR C20 M S R 3 B E A L1 L2 C1 C2 6 V2 BP BA CM
msquic vs. neqo-latest: H DC LR C20 M S R Z B :rocket:~~U~~ A L1 L2 C1 C2 6 V2 :rocket:~~BP~~ BA
mvfst vs. neqo-latest: H DC LR M 3 B L2 C2 6 BP BA
neqo vs. neqo-latest: H DC LR C20 M S R Z 3 B U E L1 L2 C1 C2 6 V2 BP BA CM
ngtcp2 vs. neqo-latest: H DC LR C20 M S R Z 3 B U E A L1 L2 C1 C2 6 V2 BP BA CM
openssl vs. neqo-latest: H DC C20 S R 3 B L2 C2 6 BP BA
picoquic vs. neqo-latest: H DC LR C20 M S R Z 3 B U E A L1 L2 C1 C2 6 V2 BP BA CM
quic-go vs. neqo-latest: H DC LR C20 M S R Z 3 B U A L1 L2 C1 C2 6 BP BA
quiche vs. neqo-latest: H DC LR M S R Z 3 B A L1 L2 C1 C2 6 BP BA
quinn vs. neqo-latest: H DC LR C20 M S R Z 3 B U E A L1 L2 C1 C2 6 BP BA
s2n-quic vs. neqo-latest: H DC LR M S R 3 B E A L1 L2 C1 C2 6 BP BA
tquic vs. neqo-latest: H DC LR M S R Z 3 B A L1 L2 C1 C2 6 BP BA
xquic vs. neqo-latest: H DC LR C20 S R Z 3 B U A L1 L2 C1 C2 6 BP BA

Unsupported Interop Tests

QUIC Interop Runner, client vs. server

neqo-latest as client

neqo-latest vs. aioquic: E CM
neqo-latest vs. go-x-net: C20 S R Z 3 E L1 C1 V2 CM
neqo-latest vs. haproxy: E CM
neqo-latest vs. kwik: E CM
neqo-latest vs. msquic: 3 E CM
neqo-latest vs. mvfst: C20 S E V2 CM
neqo-latest vs. nginx: E V2 CM
neqo-latest vs. picoquic: CM
neqo-latest vs. quic-go: E V2 CM
neqo-latest vs. quiche: E V2 CM
neqo-latest vs. quinn: V2 CM
neqo-latest vs. s2n-quic: Z V2
neqo-latest vs. tquic: E V2 CM
neqo-latest vs. xquic: S E V2 CM

neqo-latest as server

aioquic vs. neqo-latest: U E
chrome vs. neqo-latest: H DC LR C20 M S R Z B U E A L1 L2 C1 C2 6 V2 BP BA CM
go-x-net vs. neqo-latest: C20 S R Z 3 E L1 C1 V2
kwik vs. neqo-latest: E
lsquic vs. neqo-latest: Z U
msquic vs. neqo-latest: 3 E
mvfst vs. neqo-latest: C20 S R U E V2
openssl vs. neqo-latest: Z U E L1 C1 V2
quic-go vs. neqo-latest: E V2
quiche vs. neqo-latest: C20 U E V2
s2n-quic vs. neqo-latest: C20 Z U V2
tquic vs. neqo-latest: C20 U E V2
xquic vs. neqo-latest: E V2

Sep 24 '25 14:09 github-actions[bot]

Client/server transfer results

Performance differences relative to 4c741fec5396f486f7e70b81da25c1de3302e12e.

Transfer of 33554432 bytes over loopback, min. 100 runs. All unit-less numbers are in milliseconds.

Client vs. server (params)	Mean ± σ	Min	Max	MiB/s ± σ	Δ `main`	Δ `main`
google vs. google	455.2 ± 3.8	449.6	465.3	70.3 ± 8.4
google vs. neqo (cubic, paced)	277.7 ± 3.8	270.4	285.1	115.2 ± 8.4	-0.7	-0.3%
msquic vs. msquic	159.9 ± 24.4	136.2	294.0	200.1 ± 1.3
msquic vs. neqo (cubic, paced)	191.2 ± 37.3	150.9	377.4	167.4 ± 0.9	5.2	2.8%
neqo vs. google (cubic, paced)	758.9 ± 5.1	750.9	780.6	42.2 ± 6.3	:broken_heart: 2.2	0.3%
neqo vs. msquic (cubic, paced)	158.2 ± 6.9	150.8	186.4	202.2 ± 4.6	1.0	0.6%
neqo vs. neqo (cubic)	88.4 ± 4.4	82.2	105.5	361.9 ± 7.3	:green_heart: -1.7	-1.9%
neqo vs. neqo (cubic, paced)	90.1 ± 5.0	81.9	114.2	355.1 ± 6.4	:green_heart: -1.8	-1.9%
neqo vs. neqo (reno)	88.8 ± 4.4	81.5	101.3	360.3 ± 7.3	:green_heart: -2.0	-2.2%
neqo vs. neqo (reno, paced)	92.2 ± 5.1	81.1	110.3	347.2 ± 6.3	-0.9	-0.9%
neqo vs. quiche (cubic, paced)	193.4 ± 4.9	185.9	203.4	165.4 ± 6.5	0.4	0.2%
neqo vs. s2n (cubic, paced)	218.5 ± 4.0	212.6	227.5	146.4 ± 8.0	:green_heart: -2.8	-1.3%
quiche vs. neqo (cubic, paced)	153.7 ± 4.9	144.7	169.5	208.2 ± 6.5	0.6	0.4%
quiche vs. quiche	146.5 ± 5.7	137.1	173.7	218.4 ± 5.6
s2n vs. neqo (cubic, paced)	176.5 ± 5.3	166.4	190.6	181.3 ± 6.0	0.7	0.4%
s2n vs. s2n	252.1 ± 28.8	231.8	348.0	126.9 ± 1.1

Download data for profiler.firefox.com or download performance comparison data.

Sep 24 '25 14:09 github-actions[bot]

Benchmark results

Performance differences relative to 4c741fec5396f486f7e70b81da25c1de3302e12e.

1-conn/1-100mb-resp/mtu-1504 (aka. Download)/client: No change in performance detected.

       time:   [200.77 ms 201.09 ms 201.44 ms]
       thrpt:  [496.44 MiB/s 497.29 MiB/s 498.09 MiB/s]
change:
       time:   [−0.3342% −0.0566% +0.2248%] (p = 0.69 > 0.05)
       thrpt:  [−0.2243% +0.0566% +0.3353%]
Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) high severe

1-conn/10_000-parallel-1b-resp/mtu-1504 (aka. RPS)/client: No change in performance detected.

       time:   [295.03 ms 296.74 ms 298.46 ms]
       thrpt:  [33.506 Kelem/s 33.699 Kelem/s 33.895 Kelem/s]
change:
       time:   [−0.0543% +0.7568% +1.5610%] (p = 0.07 > 0.05)
       thrpt:  [−1.5370% −0.7511% +0.0543%]
Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) high mild

1-conn/1-1b-resp/mtu-1504 (aka. HPS)/client: No change in performance detected.

       time:   [28.433 ms 28.519 ms 28.626 ms]
       thrpt:  [34.933   B/s 35.064   B/s 35.171   B/s]
change:
       time:   [−0.8196% −0.2332% +0.3239%] (p = 0.43 > 0.05)
       thrpt:  [−0.3228% +0.2338% +0.8264%]
Found 19 outliers among 100 measurements (19.00%)
4 (4.00%) low severe
1 (1.00%) high mild
14 (14.00%) high severe

1-conn/1-100mb-req/mtu-1504 (aka. Upload)/client: Change within noise threshold.

       time:   [205.53 ms 205.88 ms 206.26 ms]
       thrpt:  [484.82 MiB/s 485.72 MiB/s 486.54 MiB/s]
change:
       time:   [−0.5791% −0.3358% −0.0877%] (p = 0.01 +0.3370% +0.5825%]
Found 4 outliers among 100 measurements (4.00%)
1 (1.00%) low mild
1 (1.00%) high mild
2 (2.00%) high severe

decode 4096 bytes, mask ff: No change in performance detected.

       time:   [11.604 µs 11.635 µs 11.675 µs]
       change: [−0.7613% −0.1542% +0.3640%] (p = 0.62 > 0.05)
Found 13 outliers among 100 measurements (13.00%)
1 (1.00%) low severe
4 (4.00%) low mild
8 (8.00%) high severe

decode 1048576 bytes, mask ff: No change in performance detected.

       time:   [3.0207 ms 3.0337 ms 3.0500 ms]
       change: [−0.3462% +0.1951% +0.8002%] (p = 0.53 > 0.05)
Found 10 outliers among 100 measurements (10.00%)
10 (10.00%) high severe

decode 4096 bytes, mask 7f: No change in performance detected.

       time:   [19.950 µs 20.002 µs 20.061 µs]
       change: [−0.4799% −0.0132% +0.4231%] (p = 0.96 > 0.05)
Found 19 outliers among 100 measurements (19.00%)
1 (1.00%) low severe
3 (3.00%) low mild
2 (2.00%) high mild
13 (13.00%) high severe

decode 1048576 bytes, mask 7f: No change in performance detected.

       time:   [5.0323 ms 5.0421 ms 5.0532 ms]
       change: [−0.3888% −0.0743% +0.2449%] (p = 0.65 > 0.05)
Found 12 outliers among 100 measurements (12.00%)
1 (1.00%) high mild
11 (11.00%) high severe

decode 4096 bytes, mask 3f: No change in performance detected.

       time:   [8.2931 µs 8.3342 µs 8.3792 µs]
       change: [−0.1047% +0.4086% +0.9559%] (p = 0.14 > 0.05)
Found 14 outliers among 100 measurements (14.00%)
1 (1.00%) low mild
1 (1.00%) high mild
12 (12.00%) high severe

decode 1048576 bytes, mask 3f: No change in performance detected.

       time:   [1.5871 ms 1.5939 ms 1.6020 ms]
       change: [−0.2867% +0.2766% +0.8836%] (p = 0.37 > 0.05)
Found 9 outliers among 100 measurements (9.00%)
1 (1.00%) high mild
8 (8.00%) high severe

1-streams/each-1000-bytes/wallclock-time: Change within noise threshold.

       time:   [587.36 µs 589.10 µs 591.13 µs]
       change: [−1.5026% −0.9703% −0.4296%] (p = 0.00 Found 6 outliers among 100 measurements (6.00%)
1 (1.00%) high mild
5 (5.00%) high severe
1-streams/each-1000-bytes/simulated-time
time:   [118.73 ms 118.94 ms 119.15 ms]
thrpt:  [8.1962 KiB/s 8.2105 KiB/s 8.2250 KiB/s]
change:
time:   [−0.3515% −0.1019% +0.1567%] (p = 0.42 > 0.05)
thrpt:  [−0.1565% +0.1020% +0.3527%]
No change in performance detected.
Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) high mild

1000-streams/each-1-bytes/wallclock-time: No change in performance detected.

       time:   [14.034 ms 14.060 ms 14.088 ms]
       change: [−0.5777% −0.2793% +0.0332%] (p = 0.07 > 0.05)
Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) high mild
1000-streams/each-1-bytes/simulated-time
time:   [14.991 s 15.004 s 15.017 s]
thrpt:  [66.589   B/s 66.649   B/s 66.708   B/s]
change:
time:   [−0.0763% +0.0460% +0.1680%] (p = 0.47 > 0.05)
thrpt:  [−0.1677% −0.0459% +0.0764%]
No change in performance detected.
Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) high mild

1000-streams/each-1000-bytes/wallclock-time: No change in performance detected.

       time:   [50.422 ms 50.690 ms 51.051 ms]
       change: [−0.8110% −0.1844% +0.6306%] (p = 0.64 > 0.05)
Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) high severe
1000-streams/each-1000-bytes/simulated-time
time:   [18.797 s 18.989 s 19.186 s]
thrpt:  [50.901 KiB/s 51.429 KiB/s 51.953 KiB/s]
change:
time:   [−1.0404% +0.3350% +1.8013%] (p = 0.65 > 0.05)
thrpt:  [−1.7695% −0.3339% +1.0514%]
No change in performance detected.
Found 3 outliers among 100 measurements (3.00%)
3 (3.00%) high mild

coalesce_acked_from_zero 1+1 entries: No change in performance detected.

       time:   [88.090 ns 88.405 ns 88.767 ns]
       change: [−0.6135% −0.1011% +0.4735%] (p = 0.71 > 0.05)
Found 8 outliers among 100 measurements (8.00%)
4 (4.00%) high mild
4 (4.00%) high severe

coalesce_acked_from_zero 3+1 entries: No change in performance detected.

       time:   [106.00 ns 106.33 ns 106.68 ns]
       change: [−0.9437% +0.0567% +1.2288%] (p = 0.93 > 0.05)
Found 12 outliers among 100 measurements (12.00%)
2 (2.00%) high mild
10 (10.00%) high severe

coalesce_acked_from_zero 10+1 entries: No change in performance detected.

       time:   [105.45 ns 105.97 ns 106.67 ns]
       change: [−0.3863% +0.0638% +0.5839%] (p = 0.79 > 0.05)
Found 12 outliers among 100 measurements (12.00%)
4 (4.00%) low severe
2 (2.00%) low mild
1 (1.00%) high mild
5 (5.00%) high severe

coalesce_acked_from_zero 1000+1 entries: No change in performance detected.

       time:   [88.857 ns 88.974 ns 89.112 ns]
       change: [−1.0548% +0.2316% +1.5393%] (p = 0.74 > 0.05)
Found 9 outliers among 100 measurements (9.00%)
4 (4.00%) high mild
5 (5.00%) high severe

RxStreamOrderer::inbound_frame(): Change within noise threshold.

       time:   [110.06 ms 110.21 ms 110.43 ms]
       change: [−0.8130% −0.6472% −0.4076%] (p = 0.00 Found 15 outliers among 100 measurements (15.00%)
11 (11.00%) low mild
1 (1.00%) high mild
3 (3.00%) high severe

sent::Packets::take_ranges: No change in performance detected.

       time:   [4.4441 µs 4.5158 µs 4.5739 µs]
       change: [−3.9798% −1.2144% +1.8447%] (p = 0.43 > 0.05)
Found 2 outliers among 100 measurements (2.00%)
2 (2.00%) high mild

transfer/pacing-false/varying-seeds/wallclock-time/run: No change in performance detected.

       time:   [26.754 ms 26.807 ms 26.871 ms]
       change: [−0.4325% −0.1197% +0.2200%] (p = 0.47 > 0.05)
Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) high severe

transfer/pacing-false/varying-seeds/simulated-time/run: No change in performance detected.

       time:   [25.152 s 25.189 s 25.227 s]
       thrpt:  [162.36 KiB/s 162.61 KiB/s 162.85 KiB/s]
change:
       time:   [−0.0873% +0.1284% +0.3479%] (p = 0.25 > 0.05)
       thrpt:  [−0.3467% −0.1282% +0.0874%]
Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) high mild

transfer/pacing-true/varying-seeds/wallclock-time/run: Change within noise threshold.

       time:   [27.245 ms 27.313 ms 27.383 ms]
       change: [−1.2525% −0.8901% −0.5489%] (p = 0.00 Found 2 outliers among 100 measurements (2.00%)
1 (1.00%) low mild
1 (1.00%) high mild

transfer/pacing-true/varying-seeds/simulated-time/run: No change in performance detected.

       time:   [24.942 s 24.982 s 25.024 s]
       thrpt:  [163.68 KiB/s 163.96 KiB/s 164.22 KiB/s]
change:
       time:   [−0.2610% −0.0318% +0.1851%] (p = 0.79 > 0.05)
       thrpt:  [−0.1848% +0.0318% +0.2617%]
Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) high mild

transfer/pacing-false/same-seed/wallclock-time/run: Change within noise threshold.

       time:   [26.222 ms 26.260 ms 26.310 ms]
       change: [−1.3264% −1.1389% −0.9498%] (p = 0.00 Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) high severe

transfer/pacing-false/same-seed/simulated-time/run: No change in performance detected.

       time:   [25.152 s 25.152 s 25.152 s]
       thrpt:  [162.85 KiB/s 162.85 KiB/s 162.85 KiB/s]
change:
       time:   [+0.0000% +0.0000% +0.0000%] (p = NaN > 0.05)
       thrpt:  [+0.0000% +0.0000% +0.0000%]

transfer/pacing-true/same-seed/wallclock-time/run: No change in performance detected.

       time:   [27.863 ms 27.902 ms 27.961 ms]
       change: [−0.3418% −0.0868% +0.1849%] (p = 0.55 > 0.05)
Found 4 outliers among 100 measurements (4.00%)
3 (3.00%) high mild
1 (1.00%) high severe

transfer/pacing-true/same-seed/simulated-time/run: No change in performance detected.

       time:   [25.588 s 25.588 s 25.588 s]
       thrpt:  [160.07 KiB/s 160.07 KiB/s 160.07 KiB/s]
change:
       time:   [+0.0000% +0.0000% +0.0000%] (p = NaN > 0.05)
       thrpt:  [+0.0000% +0.0000% +0.0000%]

Download data for profiler.firefox.com or download performance comparison data.

Sep 24 '25 14:09 github-actions[bot]