seanmonstar
Support for PEM-Encoded CA Certificates with Rustls Backend in reqwest Problem
Description
In my Rust project, I am using the reqwest library to make server calls over mTls, which requires adding client certificates. These certificates are in PEM format. To add them to the client builder's identity, I included the "rustls-tls" feature, which supports PEM format certificates and configured the client builder to use the rustls backend.
However, when attempting to load build-in CA certificates from the computer, it does not work as expected. The issue seems to be related to the way reqwest handles build-in CA certificates when using the rustls backend. Specifically, reqwest relies on the "rustls-tls-webpki-roots" feature to load build-in root CA certificates, but this feature is documented as supporting only DER formats, which contradicts the ability to use PEM format certificates with the rustls backend.
This inconsistency poses a challenge when working with PEM-encoded CA certificates and using the rustls backend.
Expected Behavior
I expect reqwest to support loading build-in CA certificates in PEM format when using the rustls backend, as this is a common use case for mTls.
Additional Information
The issue seems to be related to the "rustls-tls-webpki-roots" feature, which only officially supports DER formats.
Below is a code snippet of reqwesr::clientBuilde::build that used OwnedTrustAnchor::from_subject_spki_name_constraints
#[cfg(feature = "rustls-tls-webpki-roots")]
if config.tls_built_in_root_certs {
use rustls::OwnedTrustAnchor;
let trust_anchors =
webpki_roots::TLS_SERVER_ROOTS.iter().map(|trust_anchor| {
OwnedTrustAnchor::from_subject_spki_name_constraints(
trust_anchor.subject,
trust_anchor.spki,
trust_anchor.name_constraints,
)
});
root_cert_store.add_trust_anchors(trust_anchors);
}
and blow is the docuention if this function
/// Constructs an `OwnedTrustAnchor` from its components.
///
/// All inputs are DER-encoded.
///
...
Loading CA certificates in PEM format is essential for many real-world scenarios and should be supported by reqwest.
reqwest = { version = "0.11.4", features = ["blocking", "json", "stream", "rustls-tls"] } Rust version: rustc 1.73.0 (cc66ad468 2023-10-03) Operating system: windows
Example
In my case for local development, I created my own CA certificate and signed the server certificate with it, I added my CA certificate for me in the list of trusted root certificates. I launched node server with ssl with certificates issued by my CA. blow is the two options I tried to do:
#[tokio::main]
async fn main() {
let client: Client = ClientBuilder::new().build().unwrap();
let res = client.get("https://localhost:3010/api/device/descoer").send().await;
match res {
Ok(val) => {println!("{:?}", val)}
Err(err) => {println!("{:?}", err)}
}
}
the result:
Response { url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(3010), path: "/api/device/descoer", query: None, fragment: None }, status: 401, headers: {"x-powered-by": "Express", "access-control-allow-origin": "*", "content-type": "application/json; charset=utf-8", "content-length": "129", "etag": "W/\"81-4ZNFF8L66kWCN400Lh0XtKqj9W0\"", "date": "Thu, 26 Oct 2023 13:38:25 GMT", "connection": "keep-alive", "keep-alive": "timeout=5"} }
it is unauthorized because I don't attected the client certificate, but it works, is arrive to the server and get back an error
#[tokio::main]
async fn main() {
let client: Client = ClientBuilder::new().use_rustls_tls().tls_built_in_root_certs(true).build().unwrap();
let res = client.get("https://localhost:3010/api/device/descoer").send().await;
match res {
Ok(val) => {println!("{:?}", val)}
Err(err) => {println!("{:?}", err)}
}
}
and in this case is the result:
reqwest::Error { kind: Request, url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(3010), path: "/api/device/descoer", query: None, fragment: None }, source: hyper::Error(Connect, Custom { kind: Other, error: Custom { kind: InvalidData, error: InvalidCertificate(UnknownIssuer) } }) }
UnknownIssuer
Thank you for your attention to this matter. Resolving this issue would enhance the usability of reqwest for a broader range of use cases and improve the overall developer experience.
Thanks for the report! I'd welcome any contribution trying to fix this. Is it something reqwest needs to handle internally? Or is something that could be done by rustls or webpki?
It seems that this should be handled either in webpki which will allow certificates to be loaded in PEM format, or in rustls which will use something else that supports PEN format.
In any case, there is some conflict here, on the one hand, to use PEM format I must use the "rustlt backend", and the "rustle backend" itself does not support the PEM format (in the section of loading CA certificates)
This issue seems to confuse the server roots used to verify the server certificate with the client certificates (apparently stored in PEM format) that are sent to the server to verify the client's identity. The rustls project provides the rustls-pemfile crate to help parse DER out of PEM files, so you should probably consider using it.
(That is, you should probably be using some of the Identity constructors here, which do already support PEM.)
FYI a solution that worked for me was to set ClientBuilder's identity with a concatenation of cert, ca and key passed to https://docs.rs/reqwest/latest/reqwest/tls/struct.Identity.html#method.from_pem ;)
I have tried it with
- one pem file that contains certificate and the key
- one pem file that contains certificate, key and the ca
All lead to the the same error.
private key or certificate not found
Turns out Identity::from_pem() does not recognize an ENCRYPTED PRIVATE KEY, but a regular PRIVATE KEY it does understand.
This is what I had in the pem file:
-----BEGIN ENCRYPTED PRIVATE KEY-----
...
-----END ENCRYPTED PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
Another possible cause for private key or certificate not found is missing newline ("\n") in between the key and certs, if you're concatenating them on the fly (lmao)
