Lack of support for p521 signatures with the `ring`-based `reqwest/rustls` backend

[rami3l]

Rustls is completely unusable with the WARP Gateway (a corporate VPN) due to lack of support for p521 signatures.

RUSTUP_USE_RUSTLS=1 rustup update
info: syncing channel updates for 'stable-aarch64-apple-darwin'
error: could not download file from 'https://static.rust-lang.org/dist/channel-rust-stable.toml.sha256' to '~/.rustup/tmp/pnvxaiia4u2hcr_n_file'
info: syncing channel updates for 'nightly-aarch64-apple-darwin'
error: could not download file from 'https://static.rust-lang.org/dist/channel-rust-nightly.toml.sha256' to '~/.rustup/tmp/ay1l00g5xg91pnuc_file'
info: syncing channel updates for '1.63-aarch64-apple-darwin'
error: could not download file from 'https://static.rust-lang.org/dist/channel-rust-1.63.toml.sha256' to '~/.rustup/tmp/0oaqi61f4mgwqa4n_file'
info: syncing channel updates for '1.64-aarch64-apple-darwin'
error: could not download file from 'https://static.rust-lang.org/dist/channel-rust-1.64.toml.sha256' to '~/.rustup/tmp/9rhc8csclaotwleh_file'
info: syncing channel updates for '1.65-aarch64-apple-darwin'
error: could not download file from 'https://static.rust-lang.org/dist/channel-rust-1.65.toml.sha256' to '~/.rustup/tmp/26d6fm0my9i9sgvg_file'
info: checking for self-update
error: could not download file from 'https://static.rust-lang.org/rustup/release-stable.toml' to '/var/folders/lq/fqqfw_z50v96h8tlkj56c8wc0000gn/T/rustup-update5PMZuE/release-stable.toml'

Caused by:
    0: failed to make network request
    1: error sending request for url (https://static.rust-lang.org/rustup/release-stable.toml): error trying to connect: invalid peer certificate: BadSignature
    2: error trying to connect: invalid peer certificate: BadSignature
    3: invalid peer certificate: BadSignature

The curl backend has no problems with it.

Originally posted by @kornelski in https://github.com/rust-lang/rustup/issues/3806#issuecomment-2105434221

[rami3l]

@djc Can we use aws-lc instead of ring for rustls?

aws-lc seems to have p521 support already; OTOH comparing aws-lc's platform support and that of ring I can see that the former lacks mips*, but since we don't do mips* anymore (dfd71c024530779aab3665286a162b30367dfb52) this shouldn't be a problem...

I'm still not very familiar to the subject so please feel free to correct me if I'm wrong.

[djc]

Yes -- rustls should support P521 since https://github.com/rustls/rustls/releases/tag/v%2F0.22.2 if we switch to aws-lc-rs.

[rami3l]

@djc (A newbie question:) I tried adding

[dependencies]
rustls = { version = "0.22", optional = true, default-features = false, features = ["logging", "aws_lc_rs", "tls12"] }

... to our Cargo.toml but looks like ring is still there in the lockfile. Actually, both aws-lc-rs and ring are in the dependencies now, which may cause problems as described in https://github.com/seanmonstar/reqwest/pull/2225#issuecomment-2031126910.

Am I doing anything wrong, or we need to wait for something like https://github.com/seanmonstar/reqwest/issues/2136?

[kornelski]

@djc I'll try to find the reason for the p521 curve, but I'm pretty sure the choice is quite deliberate. WARP is written in Rust, so the engineers developing it feel themselves the pain of ring being incompatible with it. The ring PR is by a Cloudflare engineer.

[djc]

@kornelski I think we'll be able to use aws-lc-rs but would still be interested in hearing the reasons that it's used!

[djc]

@rami3l reqwest allows configuring the ClientBuilder with a pre-built ClientConfig (of the matching Rustls release), so I think we can build a rustls 0.22 ClientConfig and configure reqwest to use this.

[rami3l]

@rami3l reqwest allows configuring the ClientBuilder with a pre-built ClientConfig (of the matching Rustls release), so I think we can build a rustls 0.22 ClientConfig and configure reqwest to use this.

@djc That's the way to do manual resolution at runtime right? Shouldn't there be a way to simply remove ring if we're not using it?

[djc]

@rami3l reqwest allows configuring the ClientBuilder with a pre-built ClientConfig (of the matching Rustls release), so I think we can build a rustls 0.22 ClientConfig and configure reqwest to use this.

@djc That's the way to do manual resolution at runtime right? Shouldn't there be a way to simply remove ring if we're not using it?

Unfortunately it doesn't look like that exists in reqwest right now. Let's see if I can move that forward.

[kornelski]

In WARP, the p521 curve has been chosen as the best algorithm with FIPS compliance.

The p521 signature is necessary "only" to validate the root CA certificate used by WARP MITM. At the same time this is the hardest thing to change in this setup, so it's very unlikely to be changed anytime soon.

[djc]

@kornelski but in terms of impact: this specifically impacts Cloudflare's WARP deployment, right, not the default deployment one would get when setting up WARP for their organization?

(I revised https://github.com/seanmonstar/reqwest/pull/2225 yesterday to try to make progress on this.)

[kornelski]

It impacts more than just internal Cloudflare deployment. Customers have an option to upload their own CA cert (which can use any signature algorithm), but if they don't, the default Cloudflare cert is used. I don't have data on how many deployments use the incompatible cert.

[rami3l]

@kornelski but in terms of impact: this specifically impacts Cloudflare's WARP deployment, right, not the default deployment one would get when setting up WARP for their organization?

(I revised seanmonstar/reqwest#2225 yesterday to try to make progress on this.)

@djc Oops, looks like https://github.com/seanmonstar/reqwest/pull/2225 has been rejected? Should we resolve aws-lc at runtime? And if we're shipping with ring anyways, does it make sense to allow experiments on both backends? (Now it's backends relying on backends relying on backends (x_x))

[djc]

I don't think we'll need to support both aws-lc-rs and ring in rustup. I think we should use reqwest's use_preconfigured_tls() API to pass in our own ClientConfig that uses aws-lc-rs and the rustls-platform-verifier.

[rami3l]

I don't think we'll need to support both aws-lc-rs and ring in rustup. I think we should use reqwest's use_preconfigured_tls() API to pass in our own ClientConfig that uses aws-lc-rs and the rustls-platform-verifier.

@djc There's still hope that we might remove ring from the binary one day?

[djc]

There's a bunch of dead-code elimination at several stages, it's possible that ring stuff gets removed at some point anyway if we avoid using it in practice.

[rami3l]

There's a bunch of dead-code elimination at several stages, it's possible that ring stuff gets removed at some point anyway if we avoid using it in practice.

Update: Actually https://github.com/rust-lang/rustup/pull/3898 is no longer depending on ring even at compile time now.

[rami3l]

@kornelski Would you mind helping us confirm whether #3898 has worked for you?

You might be able to download the test builds from https://github.com/rust-lang/rustup/actions/runs/10192459718 (behind GitHub's login wall).

[kornelski]

@rami3l Yes, it works for me with aws_lc_rs. Thank you!