Suggestion: Use rustls/webpki-roots rather than rustls/native-tls for flexibility

[kinnison]

Problem you are trying to solve

While it doesn't affect me, I notice that a number of people have problems with Rustup, particularly in dockers or other smaller systems, when it comes to CA certificates and the like - shifting from openssl (no matter if vendored or not) to rustls/webpki-roots would allow rustup to embed the certificates needed to work cleanly, paving the way closer to a fully independent binary if desirable (ie built with musl where appropriate).

Solution you'd like

In a basic sense it ought to be as simple as:

1. Use the rustls-tls-webpki-roots feature of reqwest instead of rustls-tls-native-roots as is currently used.
2. Remove the support for rustls/default-tls

However, for full flexibility, it might also be worthwhile to add support for loading additional CA certificates from file (or indeed from the system CA store as well) so that you can still support corporate installations which need to MITM web TLS connections. This may be possibly as simple as also enabling rustls-tls-native-roots in the reqwest crate as well, in which case it might be sufficient to adjust 1. above with "as well as" in place of "instead of"

Notes

No response

[rbtcollins]

Sounds good; if someone wants to test this and put a patch up that would be great.

[ChrisDenton]

Might also want to check in with cargo folks. I suspect they have a fair amount of experience with this sort of thing.

[rami3l]

@ChrisDenton I'm dropping @weihanglo's comments on Cargo's case here:

• Allow the use of CA storing mechanisms native to the host platform [^1].
• Take care of target support: rustls depends on ring and it is possible that the latter is not supported on some lower-tier targets.
• Cargo depends on libcurl and libgit2, migrating to rustls ~~means~~ might mean they also need to depend on rustls.
• Pay attention to licensing differences between using a dylib and statically linking a lib into the binary.

[^1]: My guess would be to support corporate environments.

[djc]

As a rustls maintainer, I have a fair amount of context on the trust root subtleties.

• webpki-roots bakes trust roots into your binaries
  • Good: reliable set of trust roots from Mozilla, covering (for example) all the certificates for Rust project hosts
  • Bad: corporate trust roots for a mandatory proxy will not be included
  • Bad: roots are only upgraded over time when the binary updates, so updates (in particular, decisions to distrust) propagate slowly (or even not at all)
• rustls-native-certs finds trust roots from the operating system
  • Good: roots are updated using the usual OS mechanisms
  • Good: corporate trust roots for a mandatory proxy have a better chance of being included
  • Bad: on Windows/macOS, some (dis)trust decisions are not directly part of the included roots but are implemented as changes in the platform verifier, these decisions are not taken into account
  • Bad: Linux distributions aren't all very careful about which trust roots to include in their default bundle

A third alternative is in the works, called rustls-platform-verifier, this originates from 1Password which uses this code in their clients and has a fair bit of experience.

• rustls-platform-verifier:
  • Good: roots are updated using the usual OS mechanisms
  • Good: corporate trust roots for a mandatory proxy have a better chance of being included
  • Good: on Windows/macOS, takes (dis)trust decisions from the platform verifier into account
  • Bad: on Linux, still uses the distribution trust roots which aren't always managed carefully

My take away from this: I think rustls-platform-verifier would be a great fit for rustup, but it's not quite ready yet (even though there's a bunch of deployment experience from 1Password).

However, as explained above, I don't think using webpki-roots (via reqwest/rustls-tls-webpki-roots) only would be an improvement over using rustls-native-certs (via reqwest/rustls-tls-native-roots), because it would not be able to verify intermediate proxies that are somewhat common in corporate environments.

@ChrisDenton I'm dropping @weihanglo's comments on Cargo's case here:

* Allow the use of CA storing mechanisms native to the host platform.

* Take care of target support: `rustls` depends on [`ring`](https://github.com/briansmith/ring) and it is possible that the latter is not supported on some lower-tier targets.

The next semver-breaking rustls release will add support for pluggable cryptographic primitive backends, so this should enable more options once pure-Rust backends actually exist.

* Cargo depends  on `libcurl` and `libgit2`, migrating to `rustls` means they also need to depend on `rustls`.

rustup already uses curl today anyway, but without using rustls with it.

@rami3l maybe link to where you found those comments?

[rami3l]

@djc Oh, I got that from a Telegram chat in Chinese requesting for his comments on this issue.

rustup already uses curl today anyway, but without using rustls with it.

So when we are using the curl backend, rustup is still using the traditional, platform-specific TLS solution? It seems OK to me to have that as a fallback.

[weihanglo]

Those comments from @rami3l messaging me in private.

Cargo depends on libcurl and libgit2, migrating to rustls ~means~ might mean they also need to depend on rustls.

I was talking about TLS backend. Since rustup uses curl, this is not an issue :)

Do we need to consider the scenario that rustup is used as a library? I don't exactly know the implication though.

[djc]

Those comments from @rami3l messaging me in private.

Cargo depends on libcurl and libgit2, migrating to rustls ~means~ might mean they also need to depend on rustls.

I was talking about TLS backend. Since rustup uses curl, this is not an issue :)

Well, it can also use reqwest, and ideally we'd move to only using reqwest + rustls in the future.

Do we need to consider the scenario that rustup is used as a library? I don't exactly know the implication though.

I don't think so.

[rami3l]

Do we need to consider the scenario that rustup is used as a library? I don't exactly know the implication though.

I don't think so.

FWIW the current long-term plan seems to be having structured output or turning rustup into a daemon, instead of making it a lib: https://github.com/rust-lang/rustup/issues/3434

[djc]

Do we need to consider the scenario that rustup is used as a library? I don't exactly know the implication though.

I don't think so.

FWIW the current long-term plan seems to be having structured output or turning rustup into a daemon, instead of making it a lib: #3434

For the purpose of the current discussion, that's still more like a binary than a library. (It also seems like it would take a good while before we get there, so not sure it's worth agonizing much over at this point.)

[rami3l]

For the purpose of the current discussion, that's still more like a binary than a library. (It also seems like it would take a good while before we get there, so not sure it's worth agonizing much over at this point.)

Sorry for not being clear enough, what I want to say by mentioning that issue is exactly that rustup will very likely not become a library, and we will find other ways of exposing external APIs.

[stanal]

purpose of the current discussion, that's still more like a binary than a library. (It also seems like it would take a good while before we get there, so not sure it's worth

so it's better to use rustls-native-certs than webpki-roots now ? does the rustls-native-certs also use webpki to do verify ?

[djc]

purpose of the current discussion, that's still more like a binary than a library. (It also seems like it would take a good while before we get there, so not sure it's worth

so it's better to use rustls-native-certs than webpki-roots now ?

There are trade-offs, read my earlier comment. But, FWIW, I prefer rustls-native-certs in most of my applications.

does the rustls-native-certs also use webpki to do verify ?

rustls-native-certs crate provides a way to get the trust roots from the operating system. Typically these trust roots will be passed into a rustls::RootCertStore and used with rustls' default verifier, which used (rustls-)webpki under the hood.

[rami3l]

I'm closing this now as we're now heading to another direction (rustls-platform-verifier) in https://github.com/rust-lang/rustup/pull/3903 (so no more rustls-tls-native-roots), and I don't think an explicit transition to rustls/webpki-roots will happen any time soon with that change being made (it's the default already for WASM though, as per https://docs.rs/rustls-platform-verifier/0.3.2/rustls_platform_verifier/index.html).