neqo icon indicating copy to clipboard operation
neqo copied to clipboard
verified publisher icon mozilla

feat(transport/cc): spurious congestion event detection

Open omansfeld opened this issue 1 month ago • 12 comments

  • Adds CongestionControlStats struct with congestion_events and spurious_congestion_events stats.
  • Passes CongestionControlStats down to the congestion controller.
  • Adds mechanism to detect spurious congestion events by logging lost packet numbers in ClassicCongestionControl::maybe_lost_packets and cleaning them out as acks come in. Once all lost packets have been (late) acked a spurious congestion event is detected. The data structure is regularly purged from old packets where no late acks are expected anymore.
  • Logs those events in the added stats.

This is the first part of #2694 taking care of the detection mechanism and exposing stats. I will link a PR on the Firefox side to expose the stats as telemetry later.

The next step after this will be to implement the congestion event undo mechanism described in RFC 9438.

Some resources and other implementations of the same mechanisms:

  • quinn has very promising numbers in their PR. The graphs also nicely show the congestion window recovery.
  • msquic: This is where I got the 2*PTO maximum age for lost packets from. Packets declared longer than 2 * PTO ago are purged from the list.
  • cloudflare quiche uses a different detection mechanism where they compare lost packet count to lost packet count before the congestion event and undo if it goes below a certain threshold (due to late acks). This is more aggressive, as it doesn't make sure all lost packets were actually spurious losses.
  • it's hard to compare to what the linux kernel is doing as the detection mechanisms for TCP look very different than in QUIC, as is also described in RFC 9438 4.9.2.

Using this patch and running upload tests against [h3.speed.cloudflare.com] (RUST_LOG=neqo_bin=info cargo run --bin neqo-client -- -m=POST --stats --upload-size=300000000 https://h3.speed.cloudflare.com/) on my home network (which can generally be described as very flaky) I can actually already observe a good amount of spurious congestion events. Take e.g. this example output:

53.453 INFO stats for Client ...
  rx: 147040 drop 3 dup 0 saved 3
  tx: 215429 lost 47 lateack 22 ptoack 0 unackdrop 0            <--------- late acks observed here
  cc: congestion_events 24 spurious_congestion_events 5          <-------- spurious congestion events here
  pmtud: 4 sent 4 acked 0 lost 0 change 1500 iface_mtu 1500 pmtu
  resumed: false
  frames rx:
    crypto 6 done 1 token 0 close 0
    ack 147026 (max 215411) ping 5745 padding 0
    stream 5 reset 0 stop 0
    max: stream 0 data 224 stream_data 251
    blocked: stream 0 data 0 stream_data 0
    datagram 0
    ncid 0 rcid 0 pchallenge 0 presponse 0
    ack_frequency 0
  frames tx:
    crypto 4 done 0 token 0 close 1
    ack 5787 (max 147906) ping 4 padding 4
    stream 215250 reset 0 stop 0
    max: stream 2 data 0 stream_data 0
    blocked: stream 0 data 0 stream_data 250
    datagram 0
    ncid 0 rcid 0 pchallenge 0 presponse 0
    ack_frequency 0
  ecn:
    tx:
      Initial Count({NotEct: 4, Ect1: 0, Ect0: 0, Ce: 0})
      Handshake Count({NotEct: 4, Ect1: 0, Ect0: 0, Ce: 0})
      Short Count({NotEct: 215411, Ect1: 0, Ect0: 10, Ce: 0})
    acked:
    rx:
      Initial Count({NotEct: 4, Ect1: 0, Ect0: 0, Ce: 0})
      Handshake Count({NotEct: 3, Ect1: 0, Ect0: 0, Ce: 0})
      Short Count({NotEct: 147030, Ect1: 0, Ect0: 0, Ce: 0})
    path validation outcomes: ValidationCount({Capable: 0, NotCapable(BlackHole): 0, NotCapable(Bleaching): 1, NotCapable(ReceivedUnsentECT1): 0})
    mark transitions:
  dscp: Cs0: 147043

@mb had also observed similar spurious losses in the past when uploading to google drive and had documented his findings in #1472.

I think overall this points to the undo mechanism having big potential wins to our upload. Looking forward to see what the stats will show in the wild!

omansfeld avatar Oct 28 '25 15:10 omansfeld

Codecov Report

:x: Patch coverage is 99.52153% with 1 line in your changes missing coverage. Please review. :white_check_mark: Project coverage is 93.43%. Comparing base (6431a57) to head (f4776b5). :warning: Report is 1 commits behind head on main.

Additional details and impacted files 
@@            Coverage Diff             @@
##             main    #3086      +/-   ##
==========================================
- Coverage   93.44%   93.43%   -0.01%     
==========================================
  Files         125      125              
  Lines       36409    36568     +159     
  Branches    36409    36568     +159     
==========================================
+ Hits        34022    34168     +146     
- Misses       1542     1555      +13     
  Partials      845      845
Components Coverage Δ
neqo-common 97.36% <ø> (ø)
neqo-crypto 83.25% <ø> (-0.48%) :arrow_down:
neqo-http3 93.30% <ø> (ø)
neqo-qpack 94.29% <ø> (ø)
neqo-transport 94.55% <99.52%> (+0.03%) :arrow_up:
neqo-udp 79.42% <ø> (ø)
mtu 85.44% <ø> (ø)

codecov[bot] avatar Oct 28 '25 15:10 codecov[bot]

🐰 Bencher Report

Branchspurious_congestion_event_stats
TestbedOn-prem
Click to view all benchmark results
BenchmarkLatencyBenchmark Result
nanoseconds (ns)
(Result Δ%)		Upper Boundary
nanoseconds (ns)
(Limit %)
1-conn/1-100mb-req/mtu-1504 (aka. Upload)/client📈 view plot
🚷 view threshold		211,220,000.00 ns
(+1.85%)Baseline: 207,378,646.08 ns
216,970,652.87 ns
(97.35%)
1-conn/1-100mb-resp/mtu-1504 (aka. Download)/client📈 view plot
🚷 view threshold		204,890,000.00 ns
(+1.67%)Baseline: 201,527,790.97 ns
211,553,782.52 ns
(96.85%)
1-conn/1-1b-resp/mtu-1504 (aka. HPS)/client📈 view plot
🚷 view threshold		38,645,000.00 ns
(+17.35%)Baseline: 32,930,408.55 ns
44,915,534.72 ns
(86.04%)
1-conn/10_000-parallel-1b-resp/mtu-1504 (aka. RPS)/client📈 view plot
🚷 view threshold		285,270,000.00 ns
(-1.75%)Baseline: 290,347,790.97 ns
303,176,422.32 ns
(94.09%)
1-streams/each-1000-bytes/simulated-time📈 view plot
🚷 view threshold		118,970,000.00 ns
(+0.19%)Baseline: 118,749,168.65 ns
120,632,343.37 ns
(98.62%)
1-streams/each-1000-bytes/wallclock-time📈 view plot
🚷 view threshold		582,020.00 ns
(-1.56%)Baseline: 591,245.96 ns
613,251.05 ns
(94.91%)
1000-streams/each-1-bytes/simulated-time📈 view plot
🚷 view threshold		2,331,900,000.00 ns
(-79.68%)Baseline: 11,476,738,242.28 ns
24,723,680,121.99 ns
(9.43%)
1000-streams/each-1-bytes/wallclock-time📈 view plot
🚷 view threshold		12,637,000.00 ns
(-7.22%)Baseline: 13,620,631.83 ns
15,198,761.26 ns
(83.14%)
1000-streams/each-1000-bytes/simulated-time📈 view plot
🚷 view threshold		16,198,000,000.00 ns
(-11.22%)Baseline: 18,244,983,372.92 ns
20,967,483,132.19 ns
(77.25%)
1000-streams/each-1000-bytes/wallclock-time📈 view plot
🚷 view threshold		51,061,000.00 ns
(+0.75%)Baseline: 50,682,152.02 ns
56,497,885.93 ns
(90.38%)
RxStreamOrderer::inbound_frame()📈 view plot
🚷 view threshold		109,880,000.00 ns
(+0.14%)Baseline: 109,725,748.22 ns
111,488,315.43 ns
(98.56%)
coalesce_acked_from_zero 1+1 entries📈 view plot
🚷 view threshold		89.18 ns
(+0.21%)Baseline: 88.99 ns
90.31 ns
(98.75%)
coalesce_acked_from_zero 10+1 entries📈 view plot
🚷 view threshold		105.78 ns
(-0.26%)Baseline: 106.05 ns
107.15 ns
(98.72%)
coalesce_acked_from_zero 1000+1 entries📈 view plot
🚷 view threshold		91.65 ns
(+1.13%)Baseline: 90.63 ns
95.17 ns
(96.30%)
coalesce_acked_from_zero 3+1 entries📈 view plot
🚷 view threshold		106.23 ns
(-0.30%)Baseline: 106.55 ns
107.59 ns
(98.73%)
decode 1048576 bytes, mask 3f📈 view plot
🚷 view threshold		1,759,400.00 ns
(+6.04%)Baseline: 1,659,111.16 ns
1,858,445.90 ns
(94.67%)
decode 1048576 bytes, mask 7f📈 view plot
🚷 view threshold		5,045,400.00 ns
(-0.35%)Baseline: 5,062,949.88 ns
5,109,507.60 ns
(98.75%)
decode 1048576 bytes, mask ff📈 view plot
🚷 view threshold		2,997,000.00 ns
(-0.92%)Baseline: 3,024,775.77 ns
3,056,993.76 ns
(98.04%)
decode 4096 bytes, mask 3f📈 view plot
🚷 view threshold		6,232.20 ns
(-12.94%)Baseline: 7,158.47 ns
10,069.88 ns
(61.89%)
decode 4096 bytes, mask 7f📈 view plot
🚷 view threshold		19,643.00 ns
(-0.66%)Baseline: 19,773.05 ns
20,392.58 ns
(96.32%)
decode 4096 bytes, mask ff📈 view plot
🚷 view threshold		11,403.00 ns
(+0.34%)Baseline: 11,364.52 ns
12,411.36 ns
(91.88%)
sent::Packets::take_ranges📈 view plot
🚷 view threshold		4,590.90 ns
(-2.32%)Baseline: 4,700.02 ns
4,947.57 ns
(92.79%)
transfer/pacing-false/same-seed/simulated-time/run📈 view plot
🚷 view threshold		25,234,000,000.00 ns
(-0.57%)Baseline: 25,378,241,050.12 ns
25,959,808,020.42 ns
(97.20%)
transfer/pacing-false/same-seed/wallclock-time/run📈 view plot
🚷 view threshold		25,020,000.00 ns
(-1.96%)Baseline: 25,519,522.67 ns
27,074,045.99 ns
(92.41%)
transfer/pacing-false/varying-seeds/simulated-time/run📈 view plot
🚷 view threshold		25,223,000,000.00 ns
(+0.17%)Baseline: 25,179,293,556.09 ns
25,230,887,220.31 ns
(99.97%)
transfer/pacing-false/varying-seeds/wallclock-time/run📈 view plot
🚷 view threshold		25,224,000.00 ns
(-1.64%)Baseline: 25,645,417.66 ns
27,273,306.51 ns
(92.49%)
transfer/pacing-true/same-seed/simulated-time/run📈 view plot
🚷 view threshold		25,069,000,000.00 ns
(-1.66%)Baseline: 25,493,004,773.27 ns
26,003,308,587.97 ns
(96.41%)
transfer/pacing-true/same-seed/wallclock-time/run📈 view plot
🚷 view threshold		25,559,000.00 ns
(-4.65%)Baseline: 26,806,661.10 ns
28,669,008.07 ns
(89.15%)
transfer/pacing-true/varying-seeds/simulated-time/run📈 view plot
🚷 view threshold		25,030,000,000.00 ns
(+0.13%)Baseline: 24,996,513,126.49 ns
25,045,319,775.20 ns
(99.94%)
transfer/pacing-true/varying-seeds/wallclock-time/run📈 view plot
🚷 view threshold		25,728,000.00 ns
(-1.64%)Baseline: 26,157,644.39 ns
27,833,172.24 ns
(92.44%)
🐰 View full continuous benchmarking report in Bencher

github-actions[bot] avatar Oct 28 '25 16:10 github-actions[bot]

🐰 Bencher Report

Branchspurious_congestion_event_stats
TestbedOn-prem
Click to view all benchmark results
BenchmarkLatencyBenchmark Result
milliseconds (ms)
(Result Δ%)		Upper Boundary
milliseconds (ms)
(Limit %)
google vs. neqo (cubic, paced)📈 view plot
🚷 view threshold		278.82 ms
(+0.11%)Baseline: 278.52 ms
283.02 ms
(98.52%)
BenchmarkLatencyBenchmark Result
milliseconds (ms)
(Result Δ%)		Upper Boundary
milliseconds (ms)
(Limit %)
msquic vs. neqo (cubic, paced)📈 view plot
🚷 view threshold		221.57 ms
(+10.40%)Baseline: 200.69 ms
238.39 ms
(92.94%)
BenchmarkLatencyBenchmark Result
milliseconds (ms)
(Result Δ%)		Upper Boundary
milliseconds (ms)
(Limit %)
neqo vs. google (cubic, paced)📈 view plot
🚷 view threshold		757.79 ms
(-0.43%)Baseline: 761.10 ms
779.14 ms
(97.26%)
BenchmarkLatencyBenchmark Result
milliseconds (ms)
(Result Δ%)		Upper Boundary
milliseconds (ms)
(Limit %)
neqo vs. msquic (cubic, paced)📈 view plot
🚷 view threshold		156.41 ms
(-0.76%)Baseline: 157.61 ms
160.35 ms
(97.54%)
BenchmarkLatencyBenchmark Result
milliseconds (ms)
(Result Δ%)		Upper Boundary
milliseconds (ms)
(Limit %)
neqo vs. neqo (cubic)📈 view plot
🚷 view threshold		96.23 ms
(+4.46%)Baseline: 92.12 ms
97.83 ms
(98.37%)
BenchmarkLatencyBenchmark Result
milliseconds (ms)
(Result Δ%)		Upper Boundary
milliseconds (ms)
(Limit %)
neqo vs. neqo (cubic, paced)📈 view plot
🚷 view threshold		95.73 ms
(+2.43%)Baseline: 93.46 ms
99.03 ms
(96.67%)
BenchmarkLatencyBenchmark Result
milliseconds (ms)
(Result Δ%)		Upper Boundary
milliseconds (ms)
(Limit %)
neqo vs. neqo (reno)📈 view plot
🚷 view threshold		94.42 ms
(+2.53%)Baseline: 92.10 ms
97.67 ms
(96.68%)
BenchmarkLatencyBenchmark Result
milliseconds (ms)
(Result Δ%)		Upper Boundary
milliseconds (ms)
(Limit %)
neqo vs. neqo (reno, paced)📈 view plot
🚷 view threshold		96.56 ms
(+3.45%)Baseline: 93.34 ms
98.77 ms
(97.77%)
BenchmarkLatencyBenchmark Result
milliseconds (ms)
(Result Δ%)		Upper Boundary
milliseconds (ms)
(Limit %)
neqo vs. quiche (cubic, paced)📈 view plot
🚷 view threshold		192.77 ms
(-0.44%)Baseline: 193.62 ms
196.89 ms
(97.91%)
BenchmarkLatencyBenchmark Result
milliseconds (ms)
(Result Δ%)		Upper Boundary
milliseconds (ms)
(Limit %)
neqo vs. s2n (cubic, paced)📈 view plot
🚷 view threshold		220.92 ms
(-0.12%)Baseline: 221.19 ms
224.15 ms
(98.56%)
BenchmarkLatencyBenchmark Result
milliseconds (ms)
(Result Δ%)		Upper Boundary
milliseconds (ms)
(Limit %)
quiche vs. neqo (cubic, paced)📈 view plot
🚷 view threshold		156.72 ms
(+2.26%)Baseline: 153.26 ms
158.42 ms
(98.93%)
BenchmarkLatencyBenchmark Result
milliseconds (ms)
(Result Δ%)		Upper Boundary
milliseconds (ms)
(Limit %)
s2n vs. neqo (cubic, paced)📈 view plot
🚷 view threshold		171.78 ms
(-1.06%)Baseline: 173.63 ms
177.77 ms
(96.63%)
🐰 View full continuous benchmarking report in Bencher

github-actions[bot] avatar Oct 29 '25 13:10 github-actions[bot]

Minor comments.

Once you have been able to integrate this in Firefox and we see it working, I suggest we merge here. Sounds good?

Sounds good, I will get to the Firefox integration next week!

omansfeld avatar Oct 30 '25 14:10 omansfeld

Added a TODO for potentially tracking spurious congestion events per congestion epoch.

I have this integrated in firefox now, but not sure how to best upload the patch because it includes vendoring in the neqo changes. If it's wanted I can upload a patch to Phabricator as WIP, otherwise I'll just do the proper patch once the neqo change is in nightly.

Anyways, I can see the following in https://debug-ping-preview.firebaseapp.com/pings/ after running a few simultaneous https://h3.speed.cloudflare.com speedtests:

"networking.http_3_spurious_congestion_event_ratio": {
  "values": {
    "0": 10,
    "5000": 1 // Note the 10k scaling factor for glean metrics, so this is 50% spurious events observed for one connection
  },
  "sum": 5000
},

@mxinden can you do another review and approve if there is nothing left on your side so we can merge?

omansfeld avatar Nov 06 '25 11:11 omansfeld

#3111 got me thinking:

Should we include ECN congestion events here? They cannot be spurious, so I think it is correct to only include congestion events due to loss in the ratio recorded in glean.

Should we then also expose another stat that counts congestion events due to ECN-CE? I assume just as lost only correlates to congestion_events the amount of ECN-CE marks that we observe is also not equal to the number of congestion events caused by it. This would give us the biggest granularity and enable us to record all kind of things to glean in neqo_glue (ECN-CE vs loss ratio, spurious vs all cc events, spurious vs loss cc events, ...).

I think that would be best, but not sure if it should happen in this PR. I don't think it is much more complexity regarding review, but it's only barely related...

And while thinking about this I found an actual bug in this code: I recorded congestion events twice (in on_packets_lost and on_congestion_event). Will wait with a fix for the above discussion as it depends on what we decide here.

My suggestion would be to record loss based cc events here:

https://github.com/mozilla/neqo/blob/6d56ec96fc97b488b0d67df1e72807386fdb23dd/neqo-transport/src/cc/classic_cc.rs#L311

And ECN-CE based cc events here:

https://github.com/mozilla/neqo/blob/6d56ec96fc97b488b0d67df1e72807386fdb23dd/neqo-transport/src/cc/classic_cc.rs#L332-L334

@mxinden So hold off with the review I asked for above for now, but please share your thoughts on this when you get to it.

omansfeld avatar Nov 06 '25 15:11 omansfeld

Should we include ECN congestion events here? They cannot be spurious

Good observation! I would not include ECN when considering whether congestion events were spurious - they are a confirmed congestion event, albeit possibly at lower intensity (so might warrant a special reaction).

Should we then also expose another stat that counts congestion events due to ECN-CE?

Yes.

larseggert avatar Nov 06 '25 15:11 larseggert

Above sounds good!

For ECN CE, we currently only have:

https://glam.telemetry.mozilla.org/fog/probe/networking_http_3_ecn_ce_ect0_ratio_received/explore?

mxinden avatar Nov 06 '25 15:11 mxinden

I implemented the above in the latest commit.

We now have the stats congestion_events_due_to_ecn and congestion_events_due_to_loss and can then decide in neqo_glue on the Firefox side how exactly we want to collect those in metrics.

Also fixed the double count I had before and added a check for congestion_events_due_to_ecn counting in the existing classic_cc::tests::ecn_ce test.

If you're interested, another data point from my weird home network that sees a lot of spurious loss:

RUST_LOG=neqo_bin=info cargo run --bin neqo-client -- -m=POST --stats --upload-size=100000000 https://h3.speed.cloudflare.com/ --qlog-dir .

 tx: 72112 lost 69 lateack 41 ptoack 15 unackdrop 0
 cc: loss_congestion_events 11 ecn_congestion_events 0 spurious_congestion_events 5

And the following qvis congestion graph:

Screenshot 2025-11-06 at 16 36 32

Can really see the impact spurious congestion events have here. We basically don't spend any time in slowstart because we immediately have some spurious loss and then we're never able to really converge to a congestion point over the whole 100MB upload.

omansfeld avatar Nov 06 '25 15:11 omansfeld

Exciting to see this happening in a real network, i.e. your home network.

Did you already start the integration into Firefox? I am curious whether you see this behavior when doing normal web browsing.

I had already done the integration into firefox and tested it locally, but just from the office and not with normal web browsing. With this new version I will compile again and use it for a bit the next time I'm working from home (which might not be this week though...).

That said I addressed the comments in https://github.com/mozilla/neqo/pull/3086/commits/16dae83efe4859216648e09cb1a8e00224909e23, so I think this should be ready for another round of reviews :)

omansfeld avatar Nov 18 '25 18:11 omansfeld

CodSpeed Performance Report

Merging #3086 will degrade performances by 4.38%

Comparing omansfeld:spurious_congestion_event_stats (f4776b5) with main (6431a57)

Summary

⚡ 2 improvements
❌ 1 regression
✅ 20 untouched

:warning: Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

Mode Benchmark BASE HEAD Change
⚡ Simulation client 853.6 ms 771.9 ms +10.59%
⚡ Simulation wallclock-time 1.1 ms 1 ms +3.67%
❌ Simulation run 163 ms 170.4 ms -4.38%

codspeed-hq[bot] avatar Nov 18 '25 18:11 codspeed-hq[bot]

Failed Interop Tests

QUIC Interop Runner, client vs. server, differences relative to 5be8510e7461597c9d45449655e60265d2ff222e.

neqo-latest as client

neqo-latest as server

All results

Succeeded Interop Tests

QUIC Interop Runner, client vs. server

neqo-latest as client

neqo-latest as server

Unsupported Interop Tests

QUIC Interop Runner, client vs. server

neqo-latest as client

neqo-latest as server

github-actions[bot] avatar Nov 25 '25 14:11 github-actions[bot]

Benchmark results

Performance differences relative to 6431a57a7bbce5db5c44d40b169982a79ce08e6d.

1-conn/1-100mb-resp/mtu-1504 (aka. Download)/client: No change in performance detected.
       time:   [204.54 ms 204.89 ms 205.32 ms]
       thrpt:  [487.04 MiB/s 488.06 MiB/s 488.90 MiB/s]
change:
       time:   [-0.1005% +0.1718% +0.4459%] (p = 0.21 > 0.05)
       thrpt:  [-0.4440% -0.1715% +0.1006%]

Found 6 outliers among 100 measurements (6.00%)
3 (3.00%) low mild
2 (2.00%) high mild
1 (1.00%) high severe
1-conn/10_000-parallel-1b-resp/mtu-1504 (aka. RPS)/client: Change within noise threshold.
       time:   [283.44 ms 285.27 ms 287.16 ms]
       thrpt:  [34.824 Kelem/s 35.055 Kelem/s 35.281 Kelem/s]
change:
       time:   [+0.6622% +1.5491% +2.4770%] (p = 0.00 -1.5255% -0.6579%]

Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) high mild
1-conn/1-1b-resp/mtu-1504 (aka. HPS)/client: No change in performance detected.
       time:   [38.494 ms 38.645 ms 38.815 ms]
       thrpt:  [25.763   B/s 25.877   B/s 25.978   B/s]
change:
       time:   [-0.8712% -0.2417% +0.3695%] (p = 0.45 > 0.05)
       thrpt:  [-0.3681% +0.2423% +0.8789%]

Found 5 outliers among 100 measurements (5.00%)
1 (1.00%) high mild
4 (4.00%) high severe
1-conn/1-100mb-req/mtu-1504 (aka. Upload)/client: Change within noise threshold.
       time:   [210.96 ms 211.22 ms 211.49 ms]
       thrpt:  [472.84 MiB/s 473.44 MiB/s 474.02 MiB/s]
change:
       time:   [+0.9916% +1.1464% +1.2964%] (p = 0.00 -1.1334% -0.9819%]

Found 3 outliers among 100 measurements (3.00%)
1 (1.00%) low mild
2 (2.00%) high mild
decode 4096 bytes, mask ff: No change in performance detected.
       time:   [11.336 µs 11.403 µs 11.489 µs]
       change: [-0.2655% +0.5651% +1.4326%] (p = 0.25 > 0.05)

Found 19 outliers among 100 measurements (19.00%)
4 (4.00%) low mild
1 (1.00%) high mild
14 (14.00%) high severe
decode 1048576 bytes, mask ff: No change in performance detected.
       time:   [2.9894 ms 2.9970 ms 3.0064 ms]
       change: [-0.8768% -0.3053% +0.2327%] (p = 0.30 > 0.05)

Found 6 outliers among 100 measurements (6.00%)
6 (6.00%) high severe
decode 4096 bytes, mask 7f: No change in performance detected.
       time:   [19.588 µs 19.643 µs 19.705 µs]
       change: [-0.2540% +0.0759% +0.3907%] (p = 0.65 > 0.05)

Found 17 outliers among 100 measurements (17.00%)
2 (2.00%) low mild
2 (2.00%) high mild
13 (13.00%) high severe
decode 1048576 bytes, mask 7f: No change in performance detected.
       time:   [5.0309 ms 5.0454 ms 5.0627 ms]
       change: [-0.5423% -0.0871% +0.3420%] (p = 0.72 > 0.05)

Found 13 outliers among 100 measurements (13.00%)
13 (13.00%) high severe
decode 4096 bytes, mask 3f: No change in performance detected.
       time:   [6.2039 µs 6.2322 µs 6.2667 µs]
       change: [-0.4368% +0.4382% +1.5428%] (p = 0.44 > 0.05)

Found 14 outliers among 100 measurements (14.00%)
4 (4.00%) low mild
1 (1.00%) high mild
9 (9.00%) high severe
decode 1048576 bytes, mask 3f: No change in performance detected.
       time:   [1.7579 ms 1.7594 ms 1.7623 ms]
       change: [-0.2229% +0.0113% +0.2396%] (p = 0.83 > 0.05)

Found 5 outliers among 100 measurements (5.00%)
4 (4.00%) high mild
1 (1.00%) high severe
1-streams/each-1000-bytes/wallclock-time: No change in performance detected.
       time:   [579.64 µs 582.02 µs 584.80 µs]
       change: [-0.8642% -0.2012% +0.4908%] (p = 0.57 > 0.05)

Found 8 outliers among 100 measurements (8.00%)
8 (8.00%) high severe
1-streams/each-1000-bytes/simulated-time
time:   [118.75 ms 118.97 ms 119.18 ms]
thrpt:  [8.1938 KiB/s 8.2086 KiB/s 8.2234 KiB/s]
change:
time:   [-0.3230% -0.0640% +0.1945%] (p = 0.63 > 0.05)
thrpt:  [-0.1941% +0.0641% +0.3240%]
No change in performance detected.
1000-streams/each-1-bytes/wallclock-time: Change within noise threshold.
       time:   [12.594 ms 12.637 ms 12.681 ms]
       change: [+0.6082% +1.1776% +1.7220%] (p = 0.00
1000-streams/each-1-bytes/simulated-time: No change in performance detected.
       time:   [2.3284 s 2.3319 s 2.3354 s]
       thrpt:  [428.19   B/s 428.83   B/s 429.48   B/s]
change:
       time:   [-0.3275% -0.1033% +0.1098%] (p = 0.35 > 0.05)
       thrpt:  [-0.1097% +0.1034% +0.3286%]
1000-streams/each-1000-bytes/wallclock-time: :broken_heart: Performance has regressed.
       time:   [50.950 ms 51.061 ms 51.173 ms]
       change: [+1.9588% +2.3039% +2.6499%] (p = 0.00
1000-streams/each-1000-bytes/simulated-time: No change in performance detected.
       time:   [15.945 s 16.198 s 16.457 s]
       thrpt:  [59.340 KiB/s 60.289 KiB/s 61.247 KiB/s]
change:
       time:   [-1.8468% +0.2709% +2.5420%] (p = 0.81 > 0.05)
       thrpt:  [-2.4790% -0.2702% +1.8816%]
coalesce_acked_from_zero 1+1 entries: No change in performance detected.
       time:   [88.912 ns 89.179 ns 89.449 ns]
       change: [-0.6760% -0.2681% +0.1186%] (p = 0.19 > 0.05)

Found 6 outliers among 100 measurements (6.00%)
6 (6.00%) high mild
coalesce_acked_from_zero 3+1 entries: No change in performance detected.
       time:   [105.85 ns 106.23 ns 106.64 ns]
       change: [-0.5632% +0.0053% +0.7377%] (p = 0.99 > 0.05)

Found 13 outliers among 100 measurements (13.00%)
1 (1.00%) low mild
1 (1.00%) high mild
11 (11.00%) high severe
coalesce_acked_from_zero 10+1 entries: No change in performance detected.
       time:   [105.34 ns 105.78 ns 106.34 ns]
       change: [-0.3891% +0.0943% +0.6477%] (p = 0.74 > 0.05)

Found 11 outliers among 100 measurements (11.00%)
5 (5.00%) low mild
1 (1.00%) high mild
5 (5.00%) high severe
coalesce_acked_from_zero 1000+1 entries: No change in performance detected.
       time:   [91.560 ns 91.654 ns 91.762 ns]
       change: [-0.5207% +0.0925% +0.7317%] (p = 0.77 > 0.05)

Found 15 outliers among 100 measurements (15.00%)
5 (5.00%) high mild
10 (10.00%) high severe
RxStreamOrderer::inbound_frame(): No change in performance detected.
       time:   [109.67 ms 109.88 ms 110.18 ms]
       change: [-0.0151% +0.1956% +0.4719%] (p = 0.14 > 0.05)

Found 3 outliers among 100 measurements (3.00%)
2 (2.00%) low mild
1 (1.00%) high severe
sent::Packets::take_ranges: No change in performance detected.
       time:   [4.4755 µs 4.5909 µs 4.7040 µs]
       change: [-3.7743% -0.7540% +2.3738%] (p = 0.64 > 0.05)

Found 2 outliers among 100 measurements (2.00%)
2 (2.00%) high mild
transfer/pacing-false/varying-seeds/wallclock-time/run: Change within noise threshold.
       time:   [25.187 ms 25.224 ms 25.262 ms]
       change: [-1.1451% -0.8982% -0.6511%] (p = 0.00 Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) high mild
transfer/pacing-false/varying-seeds/simulated-time/run: No change in performance detected.
       time:   [25.185 s 25.223 s 25.261 s]
       thrpt:  [162.15 KiB/s 162.39 KiB/s 162.64 KiB/s]
change:
       time:   [-0.0440% +0.1682% +0.3869%] (p = 0.13 > 0.05)
       thrpt:  [-0.3854% -0.1679% +0.0440%]
transfer/pacing-true/varying-seeds/wallclock-time/run: Change within noise threshold.
       time:   [25.664 ms 25.728 ms 25.793 ms]
       change: [+0.3262% +0.6891% +1.0616%] (p = 0.00 Found 2 outliers among 100 measurements (2.00%)
1 (1.00%) low mild
1 (1.00%) high mild
transfer/pacing-true/varying-seeds/simulated-time/run: No change in performance detected.
       time:   [24.991 s 25.030 s 25.070 s]
       thrpt:  [163.39 KiB/s 163.65 KiB/s 163.90 KiB/s]
change:
       time:   [-0.2125% -0.0054% +0.2054%] (p = 0.96 > 0.05)
       thrpt:  [-0.2050% +0.0054% +0.2129%]

Found 4 outliers among 100 measurements (4.00%)
1 (1.00%) low mild
3 (3.00%) high mild
transfer/pacing-false/same-seed/wallclock-time/run: Change within noise threshold.
       time:   [24.992 ms 25.020 ms 25.058 ms]
       change: [+0.8796% +1.0368% +1.2116%] (p = 0.00 Found 5 outliers among 100 measurements (5.00%)
1 (1.00%) low mild
3 (3.00%) high mild
1 (1.00%) high severe
transfer/pacing-false/same-seed/simulated-time/run: No change in performance detected.
       time:   [25.234 s 25.234 s 25.234 s]
       thrpt:  [162.32 KiB/s 162.32 KiB/s 162.32 KiB/s]
change:
       time:   [+0.0000% +0.0000% +0.0000%] (p = NaN > 0.05)
       thrpt:  [+0.0000% +0.0000% +0.0000%]
transfer/pacing-true/same-seed/wallclock-time/run: Change within noise threshold.
       time:   [25.526 ms 25.559 ms 25.608 ms]
       change: [-1.6526% -1.4114% -1.1653%] (p = 0.00 Found 4 outliers among 100 measurements (4.00%)
3 (3.00%) high mild
1 (1.00%) high severe
transfer/pacing-true/same-seed/simulated-time/run: No change in performance detected.
       time:   [25.069 s 25.069 s 25.069 s]
       thrpt:  [163.39 KiB/s 163.39 KiB/s 163.39 KiB/s]
change:
       time:   [+0.0000% +0.0000% +0.0000%] (p = NaN > 0.05)
       thrpt:  [+0.0000% +0.0000% +0.0000%]

Download data for profiler.firefox.com or download performance comparison data.

github-actions[bot] avatar Nov 25 '25 14:11 github-actions[bot]

Client/server transfer results

Performance differences relative to 6431a57a7bbce5db5c44d40b169982a79ce08e6d.

Transfer of 33554432 bytes over loopback, min. 100 runs. All unit-less numbers are in milliseconds.

Client vs. server (params) Mean ± σ Min Max MiB/s ± σ Δ main Δ main
google vs. google 455.2 ± 3.9 450.4 471.5 70.3 ± 8.2
google vs. neqo (cubic, paced) 278.8 ± 4.4 269.8 294.0 114.8 ± 7.3 -0.7 -0.2%
msquic vs. msquic 191.8 ± 84.2 137.7 617.3 166.8 ± 0.4
msquic vs. neqo (cubic, paced) 221.6 ± 71.2 155.4 612.3 144.4 ± 0.4 2.5 1.2%
neqo vs. google (cubic, paced) 757.8 ± 5.0 751.0 780.7 42.2 ± 6.4 1.3 0.2%
neqo vs. msquic (cubic, paced) 156.4 ± 4.1 150.8 168.6 204.6 ± 7.8 -0.0 -0.0%
neqo vs. neqo (cubic) 96.2 ± 4.3 87.9 105.8 332.5 ± 7.4 :broken_heart: 1.9 2.0%
neqo vs. neqo (cubic, paced) 95.7 ± 4.4 87.4 104.5 334.3 ± 7.3 :green_heart: -1.9 -1.9%
neqo vs. neqo (reno) 94.4 ± 3.9 88.6 104.8 338.9 ± 8.2 -0.1 -0.2%
neqo vs. neqo (reno, paced) 96.6 ± 4.7 88.0 107.7 331.4 ± 6.8 0.8 0.8%
neqo vs. quiche (cubic, paced) 192.8 ± 4.8 185.9 207.2 166.0 ± 6.7 -0.6 -0.3%
neqo vs. s2n (cubic, paced) 220.9 ± 4.0 213.6 228.2 144.8 ± 8.0 -1.2 -0.5%
quiche vs. neqo (cubic, paced) 156.7 ± 6.8 142.9 196.0 204.2 ± 4.7 :broken_heart: 3.2 2.1%
quiche vs. quiche 146.3 ± 4.9 139.2 159.3 218.8 ± 6.5
s2n vs. neqo (cubic, paced) 171.8 ± 4.9 162.8 191.3 186.3 ± 6.5 -0.5 -0.3%
s2n vs. s2n 247.7 ± 24.8 232.6 345.3 129.2 ± 1.3

Download data for profiler.firefox.com or download performance comparison data.

github-actions[bot] avatar Nov 25 '25 14:11 github-actions[bot]