feat(transport/cc): lower CWND reduction on ECN CE vs loss
Work in progress.
Fixes https://github.com/mozilla/neqo/issues/3111.
Failed Interop Tests
QUIC Interop Runner, client vs. server, differences relative to 6d56ec96fc97b488b0d67df1e72807386fdb23dd.
neqo-latest as client
- neqo-latest vs. go-x-net: BP BA
- neqo-latest vs. haproxy: BP BA
- neqo-latest vs. kwik: BP BA
- neqo-latest vs. lsquic: :warning:L1 C1
- neqo-latest vs. msquic: R Z A L1 C1
- neqo-latest vs. mvfst: A :warning:L1 C1
- neqo-latest vs. neqo: A
- neqo-latest vs. neqo-latest: A
- neqo-latest vs. nginx: BP BA
- neqo-latest vs. ngtcp2: R Z L1 C1 CM
- neqo-latest vs. picoquic: A L1 C1
- neqo-latest vs. quic-go: A
- neqo-latest vs. quiche: BP BA
- neqo-latest vs. s2n-quic: :rocket:~~BP~~ BA CM
- neqo-latest vs. tquic: S BP BA
- neqo-latest vs. xquic: run cancelled after 20 min
neqo-latest as server
- aioquic vs. neqo-latest: CM
- go-x-net vs. neqo-latest: CM
- kwik vs. neqo-latest: BP BA CM
- msquic vs. neqo-latest: CM
- mvfst vs. neqo-latest: Z A L1 C1 CM
- neqo vs. neqo-latest: A
- openssl vs. neqo-latest: LR M A CM
- quic-go vs. neqo-latest: :warning:BP CM
- quiche vs. neqo-latest: :warning:L1 CM
- quinn vs. neqo-latest: :warning:L1 V2 CM
- s2n-quic vs. neqo-latest: CM
- tquic vs. neqo-latest: CM
- xquic vs. neqo-latest: M CM
All results
Succeeded Interop Tests
QUIC Interop Runner, client vs. server
neqo-latest as client
- neqo-latest vs. aioquic: H DC LR C20 M S R Z 3 B U A L1 L2 C1 C2 6 V2 BP BA
- neqo-latest vs. go-x-net: H DC LR M B U A L2 C2 6
- neqo-latest vs. haproxy: H DC LR C20 M S R Z 3 B U A L1 L2 C1 C2 6 V2
- neqo-latest vs. kwik: H DC LR C20 M S R Z 3 B U A L1 L2 C1 C2 6 V2
- neqo-latest vs. linuxquic: H DC LR C20 M S R Z 3 B U E A :rocket:~~L1~~ L2 :rocket:~~C1~~ C2 6 V2 BP BA CM
- neqo-latest vs. lsquic: :rocket:~~H DC LR C20 M S R Z 3 B U E A L2 C2 6 V2 BP BA CM~~
- neqo-latest vs. msquic: H DC LR C20 M S B U L2 C2 6 V2 BP BA
- neqo-latest vs. mvfst: H DC LR M R Z 3 B U :warning:L1 L2 :warning:C1 C2 6 BP BA
- neqo-latest vs. neqo: H DC LR C20 M S R Z 3 B U E L1 L2 C1 C2 6 V2 BP BA CM
- neqo-latest vs. neqo-latest: H DC LR C20 M S R Z 3 B U E L1 L2 C1 C2 6 V2 BP BA CM
- neqo-latest vs. nginx: H DC LR C20 M S R Z 3 B U A L1 L2 C1 C2 6
- neqo-latest vs. ngtcp2: H DC LR C20 M S 3 B U E A L2 C2 6 V2 BP BA
- neqo-latest vs. picoquic: H DC LR C20 M S R Z 3 B U E L2 C2 6 V2 BP BA
- neqo-latest vs. quic-go: H DC LR C20 M S R Z 3 B U L1 L2 C1 C2 6 BP BA
- neqo-latest vs. quiche: H DC LR C20 M S R Z 3 B U A L1 L2 C1 C2 6
- neqo-latest vs. quinn: H DC LR C20 M S R Z 3 B U E A L1 L2 C1 C2 6 BP BA
- neqo-latest vs. s2n-quic: H DC LR C20 M S R 3 B U E A L1 L2 C1 C2 6 :rocket:~~BP~~
- neqo-latest vs. tquic: H DC LR C20 M R Z 3 B U A L1 L2 C1 C2 6
neqo-latest as server
- aioquic vs. neqo-latest: H DC LR C20 M S R Z 3 B U A L1 L2 C1 C2 6 V2 BP BA
- chrome vs. neqo-latest: 3
- go-x-net vs. neqo-latest: H DC LR M B U A L2 C2 6 BP BA
- kwik vs. neqo-latest: H DC LR C20 M S R Z 3 B U A L1 L2 C1 C2 6 V2
- linuxquic vs. neqo-latest: H DC LR C20 M S R Z 3 B U E A L1 L2 C1 C2 6 V2 BP BA CM
- lsquic vs. neqo-latest: H DC LR C20 M S R 3 B E A L1 L2 C1 C2 6 V2 BP BA CM
- msquic vs. neqo-latest: H DC LR C20 M S R Z B U A L1 L2 C1 C2 6 V2 BP BA
- mvfst vs. neqo-latest: H DC LR M 3 B L2 C2 6 BP BA
- neqo vs. neqo-latest: H DC LR C20 M S R Z 3 B U E L1 L2 C1 C2 6 V2 BP BA CM
- ngtcp2 vs. neqo-latest: H DC LR C20 M S R Z 3 B U E A L1 L2 C1 C2 6 V2 BP BA CM
- openssl vs. neqo-latest: H DC C20 S R 3 B L2 C2 6 BP BA
- picoquic vs. neqo-latest: H DC LR C20 M S R Z 3 B U E A L1 L2 C1 C2 6 V2 BP BA CM
- quic-go vs. neqo-latest: H DC LR C20 M S R Z 3 B U A L1 L2 C1 C2 6 :warning:BP BA
- quiche vs. neqo-latest: H DC LR M S R Z 3 B A :warning:L1 L2 C1 C2 6 BP BA
- quinn vs. neqo-latest: H DC LR C20 M S R Z 3 B U E A :warning:L1 L2 C1 C2 6 BP BA
- s2n-quic vs. neqo-latest: H DC LR M S R 3 B E A L1 L2 C1 C2 6 BP BA
- tquic vs. neqo-latest: H DC LR M S R Z 3 B A L1 L2 C1 C2 6 BP BA
- xquic vs. neqo-latest: H DC LR C20 S R Z 3 B U A L1 L2 C1 C2 6 BP BA
Unsupported Interop Tests
QUIC Interop Runner, client vs. server
neqo-latest as client
- neqo-latest vs. aioquic: E CM
- neqo-latest vs. go-x-net: C20 S R Z 3 E L1 C1 V2 CM
- neqo-latest vs. haproxy: E CM
- neqo-latest vs. kwik: E CM
- neqo-latest vs. msquic: 3 E CM
- neqo-latest vs. mvfst: C20 S E V2 CM
- neqo-latest vs. nginx: E V2 CM
- neqo-latest vs. picoquic: CM
- neqo-latest vs. quic-go: E V2 CM
- neqo-latest vs. quiche: E V2 CM
- neqo-latest vs. quinn: V2 CM
- neqo-latest vs. s2n-quic: Z V2
- neqo-latest vs. tquic: E V2 CM
neqo-latest as server
- aioquic vs. neqo-latest: E
- chrome vs. neqo-latest: H DC LR C20 M S R Z B U E A L1 L2 C1 C2 6 V2 BP BA CM
- go-x-net vs. neqo-latest: C20 S R Z 3 E L1 C1 V2
- kwik vs. neqo-latest: E
- lsquic vs. neqo-latest: Z U
- msquic vs. neqo-latest: 3 E
- mvfst vs. neqo-latest: C20 S R U E V2
- openssl vs. neqo-latest: Z U E L1 C1 V2
- quic-go vs. neqo-latest: E V2
- quiche vs. neqo-latest: C20 U E V2
- s2n-quic vs. neqo-latest: C20 Z U V2
- tquic vs. neqo-latest: C20 U E V2
- xquic vs. neqo-latest: E V2
Client/server transfer results
Performance differences relative to 207d309e941d3ae9384c191c13463dc81fca4ce9.
Transfer of 33554432 bytes over loopback, min. 100 runs. All unit-less numbers are in milliseconds.
| Client vs. server (params) | Mean ± σ | Min | Max | MiB/s ± σ | Δ main |
Δ main |
|---|---|---|---|---|---|---|
| google vs. google | 455.1 ± 3.5 | 449.6 | 466.8 | 70.3 ± 9.1 | ||
| google vs. neqo (cubic, paced) | 277.2 ± 3.7 | 271.7 | 284.2 | 115.5 ± 8.6 | -0.5 | -0.2% |
| msquic vs. msquic | 204.7 ± 72.8 | 143.6 | 439.6 | 156.3 ± 0.4 | ||
| msquic vs. neqo (cubic, paced) | 218.7 ± 69.4 | 153.0 | 495.1 | 146.3 ± 0.5 | 4.3 | 2.0% |
| neqo vs. google (cubic, paced) | 758.8 ± 4.9 | 750.9 | 777.5 | 42.2 ± 6.5 | -0.2 | -0.0% |
| neqo vs. msquic (cubic, paced) | 157.7 ± 5.3 | 150.8 | 172.4 | 202.9 ± 6.0 | 0.9 | 0.6% |
| neqo vs. neqo (cubic) | 94.4 ± 4.9 | 85.5 | 108.7 | 339.0 ± 6.5 | -0.0 | -0.1% |
| neqo vs. neqo (cubic, paced) | 93.7 ± 4.0 | 88.1 | 104.5 | 341.7 ± 8.0 | :green_heart: -2.1 | -2.2% |
| neqo vs. neqo (reno) | 95.5 ± 4.0 | 85.6 | 103.1 | 335.0 ± 8.0 | 0.3 | 0.3% |
| neqo vs. neqo (reno, paced) | 94.6 ± 4.2 | 87.8 | 104.4 | 338.3 ± 7.6 | -0.1 | -0.1% |
| neqo vs. quiche (cubic, paced) | 193.6 ± 4.6 | 186.6 | 207.3 | 165.3 ± 7.0 | :broken_heart: 2.4 | 1.2% |
| neqo vs. s2n (cubic, paced) | 224.2 ± 4.3 | 214.8 | 238.1 | 142.7 ± 7.4 | :broken_heart: 2.7 | 1.2% |
| quiche vs. neqo (cubic, paced) | 150.8 ± 5.5 | 141.4 | 179.5 | 212.1 ± 5.8 | -0.8 | -0.6% |
| quiche vs. quiche | 146.3 ± 5.3 | 136.8 | 162.3 | 218.7 ± 6.0 | ||
| s2n vs. neqo (cubic, paced) | 172.6 ± 4.8 | 164.2 | 182.5 | 185.4 ± 6.7 | 1.2 | 0.7% |
| s2n vs. s2n | 251.2 ± 30.8 | 231.8 | 350.0 | 127.4 ± 1.0 |
Download data for profiler.firefox.com or download performance comparison data.
Bencher Report
| Branch | ecn-ce-reaction |
| Testbed | On-prem |
🚨 1 Alert
| Iteration | Benchmark | Measure Units | View | Benchmark Result (Result Δ%) | Upper Boundary (Limit %) |
|---|---|---|---|---|---|
| 9 | neqo vs. s2n (cubic, paced) | Latency milliseconds (ms) | 📈 plot 🚷 threshold 🚨 alert (🔔) | 224.22 ms(+1.41%)Baseline: 221.11 ms | 224.11 ms (100.05%) |
Click to view all benchmark results
| Benchmark | Latency | Benchmark Result milliseconds (ms) (Result Δ%) | Upper Boundary milliseconds (ms) (Limit %) |
|---|---|---|---|
| google vs. neqo (cubic, paced) | 📈 view plot 🚷 view threshold | 277.16 ms(-0.32%)Baseline: 278.06 ms | 281.94 ms (98.30%) |
| Benchmark | Latency | Benchmark Result milliseconds (ms) (Result Δ%) | Upper Boundary milliseconds (ms) (Limit %) |
|---|---|---|---|
| msquic vs. neqo (cubic, paced) | 📈 view plot 🚷 view threshold | 218.70 ms(+10.49%)Baseline: 197.93 ms | 235.21 ms (92.98%) |
| Benchmark | Latency | Benchmark Result milliseconds (ms) (Result Δ%) | Upper Boundary milliseconds (ms) (Limit %) |
|---|---|---|---|
| neqo vs. google (cubic, paced) | 📈 view plot 🚷 view threshold | 758.81 ms(+0.04%)Baseline: 758.55 ms | 770.35 ms (98.50%) |
| Benchmark | Latency | Benchmark Result milliseconds (ms) (Result Δ%) | Upper Boundary milliseconds (ms) (Limit %) |
|---|---|---|---|
| neqo vs. msquic (cubic, paced) | 📈 view plot 🚷 view threshold | 157.73 ms(-0.07%)Baseline: 157.84 ms | 160.77 ms (98.11%) |
| Benchmark | Latency | Benchmark Result milliseconds (ms) (Result Δ%) | Upper Boundary milliseconds (ms) (Limit %) |
|---|---|---|---|
| neqo vs. neqo (cubic) | 📈 view plot 🚷 view threshold | 94.39 ms(+3.44%)Baseline: 91.25 ms | 96.41 ms (97.91%) |
| Benchmark | Latency | Benchmark Result milliseconds (ms) (Result Δ%) | Upper Boundary milliseconds (ms) (Limit %) |
|---|---|---|---|
| neqo vs. neqo (cubic, paced) | 📈 view plot 🚷 view threshold | 93.65 ms(+1.08%)Baseline: 92.66 ms | 97.84 ms (95.72%) |
| Benchmark | Latency | Benchmark Result milliseconds (ms) (Result Δ%) | Upper Boundary milliseconds (ms) (Limit %) |
|---|---|---|---|
| neqo vs. neqo (reno) | 📈 view plot 🚷 view threshold | 95.53 ms(+4.61%)Baseline: 91.32 ms | 96.51 ms (98.99%) |
| Benchmark | Latency | Benchmark Result milliseconds (ms) (Result Δ%) | Upper Boundary milliseconds (ms) (Limit %) |
|---|---|---|---|
| neqo vs. neqo (reno, paced) | 📈 view plot 🚷 view threshold | 94.58 ms(+2.22%)Baseline: 92.52 ms | 97.44 ms (97.07%) |
| Benchmark | Latency | Benchmark Result milliseconds (ms) (Result Δ%) | Upper Boundary milliseconds (ms) (Limit %) |
|---|---|---|---|
| neqo vs. quiche (cubic, paced) | 📈 view plot 🚷 view threshold | 193.55 ms(-0.06%)Baseline: 193.67 ms | 197.03 ms (98.24%) |
| Benchmark | Latency | Benchmark Result milliseconds (ms) (Result Δ%) | Upper Boundary milliseconds (ms) (Limit %) |
|---|---|---|---|
| neqo vs. s2n (cubic, paced) | 📈 view plot 🚷 view threshold 🚨 view alert (🔔) | 224.22 ms(+1.41%)Baseline: 221.11 ms | 224.11 ms (100.05%) |
| Benchmark | Latency | Benchmark Result milliseconds (ms) (Result Δ%) | Upper Boundary milliseconds (ms) (Limit %) |
|---|---|---|---|
| quiche vs. neqo (cubic, paced) | 📈 view plot 🚷 view threshold | 150.84 ms(-1.40%)Baseline: 152.99 ms | 158.15 ms (95.38%) |
| Benchmark | Latency | Benchmark Result milliseconds (ms) (Result Δ%) | Upper Boundary milliseconds (ms) (Limit %) |
|---|---|---|---|
| s2n vs. neqo (cubic, paced) | 📈 view plot 🚷 view threshold | 172.59 ms(-0.80%)Baseline: 173.97 ms | 178.08 ms (96.92%) |
Benchmark results
Performance differences relative to 207d309e941d3ae9384c191c13463dc81fca4ce9.
1-conn/1-100mb-resp/mtu-1504 (aka. Download)/client: No change in performance detected.
time: [197.90 ms 198.31 ms 198.80 ms]
thrpt: [503.03 MiB/s 504.27 MiB/s 505.31 MiB/s]
change:
time: [−0.3444% −0.0576% +0.2452%] (p = 0.70 > 0.05)
thrpt: [−0.2446% +0.0577% +0.3455%]
Found 2 outliers among 100 measurements (2.00%)
1 (1.00%) high mild
1 (1.00%) high severe
1-conn/10_000-parallel-1b-resp/mtu-1504 (aka. RPS)/client: No change in performance detected.
time: [288.62 ms 290.19 ms 291.77 ms]
thrpt: [34.273 Kelem/s 34.460 Kelem/s 34.647 Kelem/s]
change:
time: [−0.4508% +0.3077% +1.1011%] (p = 0.44 > 0.05)
thrpt: [−1.0891% −0.3067% +0.4528%]
Found 2 outliers among 100 measurements (2.00%)
1 (1.00%) low mild
1 (1.00%) high mild
1-conn/1-1b-resp/mtu-1504 (aka. HPS)/client: No change in performance detected.
time: [38.561 ms 38.757 ms 38.971 ms]
thrpt: [25.660 B/s 25.802 B/s 25.933 B/s]
change:
time: [−0.7217% +0.0092% +0.7923%] (p = 0.98 > 0.05)
thrpt: [−0.7861% −0.0092% +0.7270%]
Found 10 outliers among 100 measurements (10.00%)
10 (10.00%) high severe
1-conn/1-100mb-req/mtu-1504 (aka. Upload)/client: :green_heart: Performance has improved.
time: [201.73 ms 202.07 ms 202.48 ms]
thrpt: [493.87 MiB/s 494.88 MiB/s 495.70 MiB/s]
change:
time: [−2.0123% −1.7715% −1.5155%] (p = 0.00 +1.8035% +2.0536%]
Found 3 outliers among 100 measurements (3.00%)
2 (2.00%) high mild
1 (1.00%) high severe
decode 4096 bytes, mask ff: No change in performance detected.
time: [10.733 µs 10.773 µs 10.819 µs]
change: [−1.0345% −0.3273% +0.3296%] (p = 0.36 > 0.05)
Found 15 outliers among 100 measurements (15.00%)
1 (1.00%) low severe
4 (4.00%) low mild
1 (1.00%) high mild
9 (9.00%) high severe
decode 1048576 bytes, mask ff: No change in performance detected.
time: [3.0247 ms 3.0378 ms 3.0544 ms]
change: [−0.3628% +0.1811% +0.8139%] (p = 0.56 > 0.05)
Found 9 outliers among 100 measurements (9.00%)
9 (9.00%) high severe
decode 4096 bytes, mask 7f: No change in performance detected.
time: [19.345 µs 19.393 µs 19.447 µs]
change: [−0.3736% +0.0620% +0.5324%] (p = 0.80 > 0.05)
Found 16 outliers among 100 measurements (16.00%)
2 (2.00%) low severe
2 (2.00%) low mild
12 (12.00%) high severe
decode 1048576 bytes, mask 7f: No change in performance detected.
time: [5.0542 ms 5.0810 ms 5.1203 ms]
change: [−0.1648% +0.4352% +1.2721%] (p = 0.26 > 0.05)
Found 17 outliers among 100 measurements (17.00%)
17 (17.00%) high severe
decode 4096 bytes, mask 3f: No change in performance detected.
time: [5.5436 µs 5.5911 µs 5.6493 µs]
change: [−0.2756% +0.4599% +1.1684%] (p = 0.24 > 0.05)
Found 14 outliers among 100 measurements (14.00%)
1 (1.00%) low mild
2 (2.00%) high mild
11 (11.00%) high severe
decode 1048576 bytes, mask 3f: No change in performance detected.
time: [1.5867 ms 1.5923 ms 1.5993 ms]
change: [−0.4410% +0.1023% +0.6410%] (p = 0.69 > 0.05)
Found 7 outliers among 100 measurements (7.00%)
1 (1.00%) high mild
6 (6.00%) high severe
1-streams/each-1000-bytes/wallclock-time: No change in performance detected.
time: [585.68 µs 586.90 µs 588.48 µs]
change: [−0.5799% −0.1201% +0.2709%] (p = 0.62 > 0.05)
Found 4 outliers among 100 measurements (4.00%)
2 (2.00%) low mild
2 (2.00%) high severe
1-streams/each-1000-bytes/simulated-time
time: [118.85 ms 119.07 ms 119.29 ms]
thrpt: [8.1864 KiB/s 8.2016 KiB/s 8.2165 KiB/s]
change:
time: [−0.2130% +0.0546% +0.3135%] (p = 0.67 > 0.05)
thrpt: [−0.3126% −0.0546% +0.2134%]
No change in performance detected.
Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) high mild
1000-streams/each-1-bytes/wallclock-time: :broken_heart: Performance has regressed.
time: [12.771 ms 12.806 ms 12.841 ms]
change: [+1.4376% +1.8690% +2.3111%] (p = 0.00 1000-streams/each-1-bytes/simulated-time: No change in performance detected.
time: [2.3328 s 2.3359 s 2.3390 s]
thrpt: [427.53 B/s 428.10 B/s 428.67 B/s]
change:
time: [−0.0939% +0.1135% +0.3278%] (p = 0.31 > 0.05)
thrpt: [−0.3267% −0.1134% +0.0940%]
1000-streams/each-1000-bytes/wallclock-time: :broken_heart: Performance has regressed.
time: [51.525 ms 51.646 ms 51.768 ms]
change: [+1.4391% +1.7474% +2.0508%] (p = 0.00 Found 2 outliers among 100 measurements (2.00%)
1 (1.00%) low mild
1 (1.00%) high mild
1000-streams/each-1000-bytes/simulated-time
time: [14.916 s 15.138 s 15.363 s]
thrpt: [63.567 KiB/s 64.509 KiB/s 65.469 KiB/s]
change:
time: [−10.896% −8.8469% −6.8007%] (p = 0.00 < 0.05)
thrpt: [+7.2970% +9.7055% +12.228%]
:green_heart: Performance has improved.coalesce_acked_from_zero 1+1 entries: No change in performance detected.
time: [89.192 ns 89.508 ns 89.819 ns]
change: [−1.4437% −0.6121% +0.0339%] (p = 0.10 > 0.05)
Found 8 outliers among 100 measurements (8.00%)
5 (5.00%) high mild
3 (3.00%) high severe
coalesce_acked_from_zero 3+1 entries: No change in performance detected.
time: [106.17 ns 106.47 ns 106.79 ns]
change: [−0.8803% −0.3134% +0.1654%] (p = 0.25 > 0.05)
Found 17 outliers among 100 measurements (17.00%)
1 (1.00%) low severe
3 (3.00%) low mild
7 (7.00%) high mild
6 (6.00%) high severe
coalesce_acked_from_zero 10+1 entries: No change in performance detected.
time: [105.59 ns 106.04 ns 106.57 ns]
change: [−0.1208% +0.4748% +1.1290%] (p = 0.14 > 0.05)
Found 15 outliers among 100 measurements (15.00%)
5 (5.00%) low mild
2 (2.00%) high mild
8 (8.00%) high severe
coalesce_acked_from_zero 1000+1 entries: No change in performance detected.
time: [91.253 ns 91.362 ns 91.490 ns]
change: [−0.7651% −0.2285% +0.2941%] (p = 0.41 > 0.05)
Found 16 outliers among 100 measurements (16.00%)
4 (4.00%) high mild
12 (12.00%) high severe
RxStreamOrderer::inbound_frame(): Change within noise threshold.
time: [109.32 ms 109.49 ms 109.76 ms]
change: [−0.8650% −0.6832% −0.3943%] (p = 0.00 Found 10 outliers among 100 measurements (10.00%)
8 (8.00%) low mild
2 (2.00%) high severesent::Packets::take_ranges: No change in performance detected.
time: [4.3933 µs 4.4903 µs 4.5814 µs]
change: [−5.5898% −2.4415% +1.0656%] (p = 0.15 > 0.05)
Found 2 outliers among 100 measurements (2.00%)
1 (1.00%) high mild
1 (1.00%) high severe
transfer/pacing-false/varying-seeds/wallclock-time/run: Change within noise threshold.
time: [24.691 ms 24.739 ms 24.790 ms]
change: [−0.9390% −0.6689% −0.4001%] (p = 0.00 Found 3 outliers among 100 measurements (3.00%)
1 (1.00%) low mild
1 (1.00%) high mild
1 (1.00%) high severetransfer/pacing-false/varying-seeds/simulated-time/run: No change in performance detected.
time: [25.194 s 25.226 s 25.259 s]
thrpt: [162.16 KiB/s 162.37 KiB/s 162.58 KiB/s]
change:
time: [−0.0657% +0.1138% +0.3064%] (p = 0.24 > 0.05)
thrpt: [−0.3055% −0.1136% +0.0657%]
Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) high mild
transfer/pacing-true/varying-seeds/wallclock-time/run: Change within noise threshold.
time: [25.719 ms 25.769 ms 25.819 ms]
change: [+0.6851% +1.0382% +1.3704%] (p = 0.00 Found 2 outliers among 100 measurements (2.00%)
2 (2.00%) high mildtransfer/pacing-true/varying-seeds/simulated-time/run: Change within noise threshold.
time: [24.788 s 24.824 s 24.861 s]
thrpt: [164.76 KiB/s 165.00 KiB/s 165.24 KiB/s]
change:
time: [−1.0559% −0.8336% −0.6281%] (p = 0.00 +0.8406% +1.0671%]
transfer/pacing-false/same-seed/wallclock-time/run: Change within noise threshold.
time: [24.858 ms 24.884 ms 24.916 ms]
change: [+1.2697% +1.4133% +1.5749%] (p = 0.00 Found 2 outliers among 100 measurements (2.00%)
1 (1.00%) high mild
1 (1.00%) high severetransfer/pacing-false/same-seed/simulated-time/run: No change in performance detected.
time: [25.234 s 25.234 s 25.234 s]
thrpt: [162.32 KiB/s 162.32 KiB/s 162.32 KiB/s]
change:
time: [+0.0000% +0.0000% +0.0000%] (p = NaN > 0.05)
thrpt: [+0.0000% +0.0000% +0.0000%]
transfer/pacing-true/same-seed/wallclock-time/run: Change within noise threshold.
time: [26.292 ms 26.326 ms 26.374 ms]
change: [−0.8637% −0.7236% −0.5429%] (p = 0.00 Found 2 outliers among 100 measurements (2.00%)
1 (1.00%) high mild
1 (1.00%) high severetransfer/pacing-true/same-seed/simulated-time/run: Change within noise threshold.
time: [25.254 s 25.254 s 25.254 s]
thrpt: [162.19 KiB/s 162.19 KiB/s 162.19 KiB/s]
change:
time: [−0.1823% −0.1823% −0.1823%] (p = 0.00 +0.1827% +0.1827%]
Download data for profiler.firefox.com or download performance comparison data.
Bencher Report
| Branch | ecn-ce-reaction |
| Testbed | On-prem |
🚨 1 Alert
| Benchmark | Measure Units | View | Benchmark Result (Result Δ%) | Upper Boundary (Limit %) |
|---|---|---|---|---|
| transfer/pacing-false/varying-seeds/simulated-time/run | Latency seconds (s) | 📈 plot 🚷 threshold 🚨 alert (🔔) | 25.23 s(+0.21%)Baseline: 25.17 s | 25.22 s (100.02%) |
Click to view all benchmark results
| Benchmark | Latency | Benchmark Result nanoseconds (ns) (Result Δ%) | Upper Boundary nanoseconds (ns) (Limit %) |
|---|---|---|---|
| 1-conn/1-100mb-req/mtu-1504 (aka. Upload)/client | 📈 view plot 🚷 view threshold | 202,070,000.00 ns(-2.34%)Baseline: 206,904,153.35 ns | 216,801,155.68 ns (93.21%) |
| 1-conn/1-100mb-resp/mtu-1504 (aka. Download)/client | 📈 view plot 🚷 view threshold | 198,310,000.00 ns(-1.32%)Baseline: 200,959,297.12 ns | 211,660,162.93 ns (93.69%) |
| 1-conn/1-1b-resp/mtu-1504 (aka. HPS)/client | 📈 view plot 🚷 view threshold | 38,757,000.00 ns(+25.28%)Baseline: 30,935,504.79 ns | 41,369,058.31 ns (93.69%) |
| 1-conn/10_000-parallel-1b-resp/mtu-1504 (aka. RPS)/client | 📈 view plot 🚷 view threshold | 290,190,000.00 ns(-0.50%)Baseline: 291,658,306.71 ns | 304,189,804.54 ns (95.40%) |
| 1-streams/each-1000-bytes/simulated-time | 📈 view plot 🚷 view threshold | 119,070,000.00 ns(+0.37%)Baseline: 118,635,047.92 ns | 120,751,498.91 ns (98.61%) |
| 1-streams/each-1000-bytes/wallclock-time | 📈 view plot 🚷 view threshold | 586,900.00 ns(-1.07%)Baseline: 593,275.91 ns | 616,667.41 ns (95.17%) |
| 1000-streams/each-1-bytes/simulated-time | 📈 view plot 🚷 view threshold | 2,335,900,000.00 ns(-84.04%)Baseline: 14,631,772,843.45 ns | 19,580,063,385.77 ns (11.93%) |
| 1000-streams/each-1-bytes/wallclock-time | 📈 view plot 🚷 view threshold | 12,806,000.00 ns(-8.18%)Baseline: 13,947,313.10 ns | 14,967,389.73 ns (85.56%) |
| 1000-streams/each-1000-bytes/simulated-time | 📈 view plot 🚷 view threshold | 15,138,000,000.00 ns(-19.85%)Baseline: 18,887,000,000.00 ns | 19,984,377,245.14 ns (75.75%) |
| 1000-streams/each-1000-bytes/wallclock-time | 📈 view plot 🚷 view threshold | 51,646,000.00 ns(+2.00%)Baseline: 50,633,900.96 ns | 57,342,790.67 ns (90.07%) |
| RxStreamOrderer::inbound_frame() | 📈 view plot 🚷 view threshold | 109,490,000.00 ns(-0.18%)Baseline: 109,692,907.35 ns | 111,642,666.18 ns (98.07%) |
| coalesce_acked_from_zero 1+1 entries | 📈 view plot 🚷 view threshold | 89.51 ns(+0.82%)Baseline: 88.78 ns | 89.88 ns (99.59%) |
| coalesce_acked_from_zero 10+1 entries | 📈 view plot 🚷 view threshold | 106.04 ns(-0.01%)Baseline: 106.05 ns | 107.15 ns (98.96%) |
| coalesce_acked_from_zero 1000+1 entries | 📈 view plot 🚷 view threshold | 91.36 ns(+1.39%)Baseline: 90.11 ns | 94.58 ns (96.60%) |
| coalesce_acked_from_zero 3+1 entries | 📈 view plot 🚷 view threshold | 106.47 ns(-0.09%)Baseline: 106.56 ns | 107.61 ns (98.94%) |
| decode 1048576 bytes, mask 3f | 📈 view plot 🚷 view threshold | 1,592,300.00 ns(-2.04%)Baseline: 1,625,532.91 ns | 1,785,839.32 ns (89.16%) |
| decode 1048576 bytes, mask 7f | 📈 view plot 🚷 view threshold | 5,081,000.00 ns(+0.31%)Baseline: 5,065,317.89 ns | 5,110,024.98 ns (99.43%) |
| decode 1048576 bytes, mask ff | 📈 view plot 🚷 view threshold | 3,037,800.00 ns(+0.28%)Baseline: 3,029,216.29 ns | 3,047,001.99 ns (99.70%) |
| decode 4096 bytes, mask 3f | 📈 view plot 🚷 view threshold | 5,591.10 ns(-24.18%)Baseline: 7,374.61 ns | 10,407.17 ns (53.72%) |
| decode 4096 bytes, mask 7f | 📈 view plot 🚷 view threshold | 19,393.00 ns(-2.08%)Baseline: 19,804.79 ns | 20,474.78 ns (94.72%) |
| decode 4096 bytes, mask ff | 📈 view plot 🚷 view threshold | 10,773.00 ns(-5.19%)Baseline: 11,362.27 ns | 12,549.26 ns (85.85%) |
| sent::Packets::take_ranges | 📈 view plot 🚷 view threshold | 4,490.30 ns(-5.02%)Baseline: 4,727.68 ns | 4,965.51 ns (90.43%) |
| transfer/pacing-false/same-seed/simulated-time/run | 📈 view plot 🚷 view threshold | 25,234,000,000.00 ns(-0.76%)Baseline: 25,428,331,189.71 ns | 26,063,680,149.37 ns (96.82%) |
| transfer/pacing-false/same-seed/wallclock-time/run | 📈 view plot 🚷 view threshold | 24,884,000.00 ns(-3.45%)Baseline: 25,772,376.21 ns | 27,088,343.66 ns (91.86%) |
| transfer/pacing-false/varying-seeds/simulated-time/run | 📈 view plot 🚷 view threshold 🚨 view alert (🔔) | 25,226,000,000.00 ns(+0.21%)Baseline: 25,173,643,086.82 ns | 25,221,994,030.43 ns (100.02%) |
| transfer/pacing-false/varying-seeds/wallclock-time/run | 📈 view plot 🚷 view threshold | 24,739,000.00 ns(-4.23%)Baseline: 25,831,672.03 ns | 27,473,592.18 ns (90.05%) |
| transfer/pacing-true/same-seed/simulated-time/run | 📈 view plot 🚷 view threshold | 25,254,000,000.00 ns(-1.37%)Baseline: 25,605,781,350.48 ns | 25,844,084,982.26 ns (97.72%) |
| transfer/pacing-true/same-seed/wallclock-time/run | 📈 view plot 🚷 view threshold | 26,326,000.00 ns(-2.85%)Baseline: 27,098,450.16 ns | 28,642,091.89 ns (91.91%) |
| transfer/pacing-true/varying-seeds/simulated-time/run | 📈 view plot 🚷 view threshold | 24,824,000,000.00 ns(-0.68%)Baseline: 24,993,520,900.32 ns | 25,046,383,089.07 ns (99.11%) |
| transfer/pacing-true/varying-seeds/wallclock-time/run | 📈 view plot 🚷 view threshold | 25,769,000.00 ns(-2.15%)Baseline: 26,334,472.67 ns | 28,061,333.28 ns (91.83%) |
@omansfeld finished this in https://github.com/mozilla/neqo/pull/3233. :pray: