neqo icon indicating copy to clipboard operation
neqo copied to clipboard

Published 20 hours ago • verified publisher icon mozilla

ci: Use `vmactions` for checks on other platforms

Open larseggert opened this issue 1 year ago • 3 comments

WIP

This needs to wait until the nss packages for these platforms have caught up with our minimum version requirement, since building NSS on these platforms requires extensive patching.

As of 2024-10-3:

  • [ ] FreeBSD: https://www.freshports.org/security/nss/
  • [ ] NetBSD: https://ftp.netbsd.org/pub/pkgsrc/packages/NetBSD/amd64/10.0/All/
  • [ ] OpenBSD: https://cdn.openbsd.org/pub/OpenBSD/7.6/packages/amd64/

Ditto for their rust packages, since rustup doesn't support most of these.

larseggert avatar Sep 24 '24 10:09 larseggert

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 93.32%. Comparing base (763ac2c) to head (a8d7e8c). Report is 12 commits behind head on main.

Additional details and impacted files 
@@            Coverage Diff             @@
##             main    #2133      +/-   ##
==========================================
- Coverage   93.33%   93.32%   -0.01%     
==========================================
  Files         114      114              
  Lines       36887    36889       +2     
  Branches    36887    36889       +2     
==========================================
  Hits        34428    34428              
- Misses       1679     1681       +2     
  Partials      780      780

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov[bot] avatar Sep 24 '24 11:09 codecov[bot]

Failed Interop Tests

QUIC Interop Runner, client vs. server, differences relative to 6013bde7983b9cc77939b0c084ba47dcbb8478ba.

neqo-latest as client

neqo-latest as server

All results

Succeeded Interop Tests

QUIC Interop Runner, client vs. server

neqo-latest as client

neqo-latest as server

Unsupported Interop Tests

QUIC Interop Runner, client vs. server

neqo-latest as client

neqo-latest as server

github-actions[bot] avatar Sep 24 '24 13:09 github-actions[bot]

Benchmark results

Performance differences relative to 6013bde7983b9cc77939b0c084ba47dcbb8478ba.

decode 4096 bytes, mask ff: No change in performance detected.
       time:   [11.137 µs 11.153 µs 11.179 µs]
       change: [-0.3835% -0.1140% +0.1579%] (p = 0.42 > 0.05)

Found 14 outliers among 100 measurements (14.00%)
3 (3.00%) low severe
4 (4.00%) low mild
3 (3.00%) high mild
4 (4.00%) high severe
decode 1048576 bytes, mask ff: No change in performance detected.
       time:   [3.0167 ms 3.0259 ms 3.0365 ms]
       change: [-0.5709% -0.0869% +0.3625%] (p = 0.73 > 0.05)

Found 10 outliers among 100 measurements (10.00%)
10 (10.00%) high severe
decode 4096 bytes, mask 7f: No change in performance detected.
       time:   [19.494 µs 19.543 µs 19.605 µs]
       change: [-1.1043% -0.4302% +0.0660%] (p = 0.17 > 0.05)

Found 15 outliers among 100 measurements (15.00%)
1 (1.00%) low severe
5 (5.00%) low mild
1 (1.00%) high mild
8 (8.00%) high severe
decode 1048576 bytes, mask 7f: No change in performance detected.
       time:   [5.1634 ms 5.1876 ms 5.2241 ms]
       change: [-0.3159% +0.2212% +0.9739%] (p = 0.55 > 0.05)

Found 16 outliers among 100 measurements (16.00%)
1 (1.00%) high mild
15 (15.00%) high severe
decode 4096 bytes, mask 3f: No change in performance detected.
       time:   [5.5266 µs 5.5502 µs 5.5791 µs]
       change: [-0.8138% +0.8606% +3.5885%] (p = 0.63 > 0.05)

Found 12 outliers among 100 measurements (12.00%)
4 (4.00%) low mild
1 (1.00%) high mild
7 (7.00%) high severe
decode 1048576 bytes, mask 3f: No change in performance detected.
       time:   [1.7598 ms 1.7652 ms 1.7720 ms]
       change: [-0.4735% -0.0003% +0.4711%] (p = 0.99 > 0.05)

Found 11 outliers among 100 measurements (11.00%)
4 (4.00%) high mild
7 (7.00%) high severe
coalesce_acked_from_zero 1+1 entries: No change in performance detected.
       time:   [98.783 ns 99.098 ns 99.422 ns]
       change: [-0.3303% +0.0867% +0.4848%] (p = 0.68 > 0.05)

Found 14 outliers among 100 measurements (14.00%)
6 (6.00%) high mild
8 (8.00%) high severe
coalesce_acked_from_zero 3+1 entries: No change in performance detected.
       time:   [116.59 ns 116.95 ns 117.34 ns]
       change: [-0.8583% +0.0112% +0.7011%] (p = 0.98 > 0.05)

Found 16 outliers among 100 measurements (16.00%)
2 (2.00%) low mild
1 (1.00%) high mild
13 (13.00%) high severe
coalesce_acked_from_zero 10+1 entries: No change in performance detected.
       time:   [116.30 ns 116.73 ns 117.25 ns]
       change: [-0.4579% +0.2334% +0.8587%] (p = 0.51 > 0.05)

Found 12 outliers among 100 measurements (12.00%)
1 (1.00%) low severe
4 (4.00%) low mild
7 (7.00%) high severe
coalesce_acked_from_zero 1000+1 entries: No change in performance detected.
       time:   [97.800 ns 97.954 ns 98.135 ns]
       change: [-1.0089% +0.2774% +1.5403%] (p = 0.69 > 0.05)

Found 16 outliers among 100 measurements (16.00%)
7 (7.00%) high mild
9 (9.00%) high severe
RxStreamOrderer::inbound_frame(): No change in performance detected.
       time:   [111.09 ms 111.22 ms 111.43 ms]
       change: [-0.1373% +0.1060% +0.3182%] (p = 0.45 > 0.05)

Found 21 outliers among 100 measurements (21.00%)
1 (1.00%) low severe
9 (9.00%) low mild
10 (10.00%) high mild
1 (1.00%) high severe
SentPackets::take_ranges: No change in performance detected.
       time:   [5.4235 µs 5.5244 µs 5.6413 µs]
       change: [-1.6863% +0.4755% +2.8436%] (p = 0.69 > 0.05)

Found 5 outliers among 100 measurements (5.00%)
3 (3.00%) high mild
2 (2.00%) high severe
transfer/pacing-false/varying-seeds: Change within noise threshold.
       time:   [42.144 ms 42.224 ms 42.306 ms]
       change: [+0.9208% +1.2048% +1.4981%] (p = 0.00 Found 2 outliers among 100 measurements (2.00%)
1 (1.00%) low mild
1 (1.00%) high mild
transfer/pacing-true/varying-seeds: Change within noise threshold.
       time:   [42.427 ms 42.500 ms 42.574 ms]
       change: [+0.9057% +1.1443% +1.3898%] (p = 0.00 Found 4 outliers among 100 measurements (4.00%)
4 (4.00%) high mild
transfer/pacing-false/same-seed: Change within noise threshold.
       time:   [42.178 ms 42.239 ms 42.301 ms]
       change: [+0.7686% +1.0066% +1.2432%] (p = 0.00 Found 3 outliers among 100 measurements (3.00%)
1 (1.00%) low mild
2 (2.00%) high mild
transfer/pacing-true/same-seed: Change within noise threshold.
       time:   [42.715 ms 42.784 ms 42.860 ms]
       change: [+1.3737% +1.6203% +1.8716%] (p = 0.00 Found 2 outliers among 100 measurements (2.00%)
1 (1.00%) low mild
1 (1.00%) high severe
1-conn/1-100mb-resp/mtu-1504 (aka. Download)/client: No change in performance detected.
       time:   [877.23 ms 887.16 ms 897.33 ms]
       thrpt:  [111.44 MiB/s 112.72 MiB/s 113.99 MiB/s]
change:
       time:   [-1.5464% +0.0297% +1.5359%] (p = 0.97 > 0.05)
       thrpt:  [-1.5126% -0.0297% +1.5707%]
1-conn/10_000-parallel-1b-resp/mtu-1504 (aka. RPS)/client: No change in performance detected.
       time:   [300.72 ms 303.02 ms 305.37 ms]
       thrpt:  [32.748 Kelem/s 33.002 Kelem/s 33.253 Kelem/s]
change:
       time:   [-0.8400% +0.2123% +1.2972%] (p = 0.69 > 0.05)
       thrpt:  [-1.2806% -0.2119% +0.8472%]

Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) high mild
1-conn/1-1b-resp/mtu-1504 (aka. HPS)/client: No change in performance detected.
       time:   [34.375 ms 34.574 ms 34.792 ms]
       thrpt:  [28.742  elem/s 28.924  elem/s 29.091  elem/s]
change:
       time:   [-0.0851% +0.7820% +1.6247%] (p = 0.07 > 0.05)
       thrpt:  [-1.5988% -0.7759% +0.0852%]

Found 9 outliers among 100 measurements (9.00%)
1 (1.00%) low mild
8 (8.00%) high severe
1-conn/1-100mb-resp/mtu-1504 (aka. Upload)/client: Change within noise threshold.
       time:   [1.6073 s 1.6220 s 1.6367 s]
       thrpt:  [61.100 MiB/s 61.654 MiB/s 62.216 MiB/s]
change:
       time:   [-3.6381% -2.2491% -0.8102%] (p = 0.00 +2.3008% +3.7755%]

Client/server transfer results

Transfer of 33554432 bytes over loopback.

Client Server CC Pacing MTU Mean [ms] Min [ms] Max [ms]
gquiche gquiche 1504 585.0 ± 96.7 504.5 763.2
neqo gquiche reno on 1504 803.4 ± 88.6 755.1 1008.3
neqo gquiche reno 1504 787.0 ± 39.6 758.4 890.8
neqo gquiche cubic on 1504 806.2 ± 83.8 754.2 975.5
neqo gquiche cubic 1504 786.0 ± 82.5 721.6 985.4
msquic msquic 1504 191.1 ± 125.6 99.2 613.9
neqo msquic reno on 1504 297.5 ± 103.9 209.8 492.8
neqo msquic reno 1504 270.6 ± 90.0 207.9 460.0
neqo msquic cubic on 1504 288.9 ± 95.3 212.6 469.3
neqo msquic cubic 1504 245.6 ± 64.7 205.4 425.4
gquiche neqo reno on 1504 712.3 ± 122.6 553.8 895.8
gquiche neqo reno 1504 693.9 ± 94.6 559.1 862.0
gquiche neqo cubic on 1504 704.7 ± 114.8 568.7 888.5
gquiche neqo cubic 1504 718.5 ± 114.1 560.7 922.5
msquic neqo reno on 1504 511.8 ± 62.1 471.5 629.3
msquic neqo reno 1504 490.5 ± 58.6 461.5 656.5
msquic neqo cubic on 1504 489.4 ± 37.6 461.1 591.3
msquic neqo cubic 1504 515.5 ± 53.8 469.2 602.5
neqo neqo reno on 1504 535.7 ± 70.0 451.2 679.3
neqo neqo reno 1504 537.6 ± 53.5 480.9 632.5
neqo neqo cubic on 1504 551.2 ± 49.6 463.3 643.4
neqo neqo cubic 1504 552.0 ± 32.1 469.4 583.5

:arrow_down: Download logs

github-actions[bot] avatar Sep 24 '24 13:09 github-actions[bot]

@mxinden is the test failure legit? https://github.com/mozilla/neqo/actions/runs/12704862858/job/35414911552?pr=2133#step:5:3588

larseggert avatar Jan 10 '25 07:01 larseggert

Thanks for the ping @larseggert. Failure should be fixed with https://github.com/mozilla/neqo/pull/2339.

mxinden avatar Jan 10 '25 08:01 mxinden

FreeBSD now works, OpenBSD and Solaris have too old NSS packages.

On NetBSD, test_fixture::fixture_init() fails with The certificate/key database is in an old, unsupported format., which I don't quite know what to do about. CC @martinthomson for when you're back.

larseggert avatar Jan 10 '25 08:01 larseggert

I assume there is no easy way to reduce some of the repetition?

We could move the scripts out of the .yml into a shared .sh, but that would then need to differentiate between the platforms. Or maybe we can dynamically build the script with workflow steps before passing it to the vmaction step for execution?

On a more general note, do I understand correctly that we are supporting these platforms on a best effort basis only?

Totally. This is mostly to see if we can easily get things working for them.

larseggert avatar Jan 10 '25 10:01 larseggert