cloudflare
Feat: Server TLS Certificate bundle + SNI based resolver
What is the problem your feature solves, or the need it fulfills?
First of all, I started taking a look at pingora over the weekend and it looks awesome! Thanks so much for open-sourcing it!
What I want to achieve (and already did during testing) is to have the ability to provide multiple TLS certificates and make it possible to automatically choose the correct certificate based on the SNI from the ClientHello.
This would make it possible to use pingora as a main gateway with a single IP while combining multiple different domains. For instance, if you only have a single IP, you can serve literally anything from your backend purely based on SNI. As soon as then #567 is merged, this combination will be super powerful and you can not only choose upstream servers based on SNI, but also serve different certificates.
Describe the solution you'd like
I already implemented a working solution based on rustls, but I did not open a PR out of the blue, because I wanted to check back with you, if you would be open for something like this. It did not need that many changes and it actually works fine after local testing. You can take a look at the branch here or I could also open a PR from this.
Basically, what I did is I changed pingora-core/src/listeners/tls/rustls/mod.rs::TlsSettings and made it possible to select either a single cert + key (like it is now by default), or optionally provide a certificate + key + SNI bundle, which then will create an SNI based Resolver.
Describe alternatives you've considered
Since pingora did not make it possible by default (at least not for rustls, don't know about openssl), there is no real alternative I would be aware of.
Additional context
Since I already have a working example locally, let me show you what I can do with it.
I did not change the current API on purpose to not introduce unnecessary breaking changes. You can still do
TlsSettings::intermediate(&tls_cert, &tls_key).unwrap();
But, what is also possible now is the following, to provide any number of certificates:
let certs = vec![
BundleCert {
sni: "test1.local".to_string(),
cert_path: "tls/test1/cert-chain.pem".to_string(),
key_path: "tls/test1/key.pem".to_string(),
},
BundleCert {
sni: "test2.local".to_string(),
cert_path: "tls/test2/cert-chain.pem".to_string(),
key_path: "tls/test2/key.pem".to_string(),
},
];
let mut tls_settings = TlsSettings::intermediate_bundle(certs).unwrap();
This will now create a rustls::server::ResolvesServerCertUsingSni which adds all the provided bundle Certificates. Depending on the SNI from the ClientHello, a matching a certificate will be chosen automatically. This works nicely:
❯ openssl x509 -noout -ext subjectAltName -in tls/test1/cert-chain.pem
X509v3 Subject Alternative Name:
DNS:localhost, DNS:test1.local
❯ openssl x509 -noout -ext subjectAltName -in tls/test2/cert-chain.pem
X509v3 Subject Alternative Name:
DNS:localhost, DNS:test2.local
Which then makes the handshakes work for both DNS names
test1.local:
❯ openssl s_client -connect localhost:8080 -servername test1.local
Connecting to 127.0.0.1
CONNECTED(00000003)
depth=2 CN=Nioca Root
verify error:num=19:self-signed certificate in certificate chain
verify return:1
depth=2 CN=Nioca Root
verify return:1
depth=1 CN=Nioca Intermediate
verify return:1
depth=0 CN=localhost, O=rerox test
verify return:1
---
Certificate chain
0 s:CN=localhost, O=rerox test
i:CN=Nioca Intermediate
a:PKEY: id-ecPublicKey, 384 (bit); sigalg: ecdsa-with-SHA384
v:NotBefore: Apr 27 13:16:21 2025 GMT; NotAfter: May 7 13:26:21 2026 GMT
1 s:CN=Nioca Intermediate
i:CN=Nioca Root
a:PKEY: id-ecPublicKey, 384 (bit); sigalg: ecdsa-with-SHA384
v:NotBefore: May 7 16:42:41 2024 GMT; NotAfter: Apr 24 18:52:41 2035 GMT
2 s:CN=Nioca Root
i:CN=Nioca Root
a:PKEY: id-ecPublicKey, 384 (bit); sigalg: ecdsa-with-SHA384
v:NotBefore: May 20 21:04:32 2023 GMT; NotAfter: Apr 19 18:52:32 2055 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIB/zCCAYWgAwIBAgIBAjAKBggqhkjOPQQDAzAdMRswGQYDVQQDDBJOaW9jYSBJ
bnRlcm1lZGlhdGUwHhcNMjUwNDI3MTMxNjIxWhcNMjYwNTA3MTMyNjIxWjApMRIw
EAYDVQQDDAlsb2NhbGhvc3QxEzARBgNVBAoMCnJlcm94IHRlc3QwdjAQBgcqhkjO
PQIBBgUrgQQAIgNiAASIPiRDvYy0SFQ65fop0JquBduPJK9aWflJ8SHUXZe+0PJu
fr0A143cEszWliRL9V6y5K6Tms8lZ5RQPlgY7qfSCjEGZpNGReMCxQe0akwwozZE
5wfIUa/a3amkWg9QiVyjgYwwgYkwHwYDVR0jBBgwFoAUK/pd61lzdkNyi8AcSE8P
ZWpnCzQwIQYDVR0RBBowGIIJbG9jYWxob3N0ggt0ZXN0MS5sb2NhbDATBgNVHSUE
DDAKBggrBgEFBQcDATAdBgNVHQ4EFgQUCmWGZ/YZpMJxX+AqhidGvptralYwDwYD
VR0TAQH/BAUwAwEBADAKBggqhkjOPQQDAwNoADBlAjAMs621n/TVTHidQ6JcPJTD
R+nWbK4zH0XcU30ORFmd/BzgDaLIiCjKS/KkDW1nz8ECMQC+O9QszgLHk/7WbNLu
d2h6QlsgoeE/68DYVlhjxtujsMbbMc1K8DqNQtdkk5Iu5ZQ=
-----END CERTIFICATE-----
subject=CN=localhost, O=rerox test
issuer=CN=Nioca Intermediate
---
No client certificate CA names sent
Peer signing digest: SHA384
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1811 bytes and written 402 bytes
Verification error: self-signed certificate in certificate chain
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 384 bit
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 19 (self-signed certificate in certificate chain)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
Protocol : TLSv1.3
Cipher : TLS_AES_256_GCM_SHA384
Session-ID: B4D119EF274DDA5D07F12BAF11D155B9A6149C26A7FEDDDA280283EAD4A737BA
Session-ID-ctx:
Resumption PSK: 0B8B5D6F040F79E5188FB102B0FF1FAC0241E9389420E852710C2CCA6D98532AD9C94F6B0E7C4858A4171556D3FEC373
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 86400 (seconds)
TLS session ticket:
0000 - 9e a0 2a 48 91 93 f7 29-e0 81 e4 95 b8 bf c9 69 ..*H...).......i
0010 - c6 02 ff da 47 49 a1 c0-9f a3 4c 0c a9 32 c9 fc ....GI....L..2..
Start Time: 1745761853
Timeout : 7200 (sec)
Verify return code: 19 (self-signed certificate in certificate chain)
Extended master secret: no
Max Early Data: 0
---
read R BLOCK
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
Protocol : TLSv1.3
Cipher : TLS_AES_256_GCM_SHA384
Session-ID: FF0D7059CF7F09DE44F73450ED3147E67CF1E46C6AF2796031896ACD6F90A844
Session-ID-ctx:
Resumption PSK: 12BE27EFC34B867B616A9E6B8DD200EEB9FF6C2C5A58D0D50CDC286A9DE38F395B5A04B2CE14B078DA334E41BF744543
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 86400 (seconds)
TLS session ticket:
0000 - 62 73 bf 24 98 79 8d fa-ee 17 55 f8 73 cd 1c 40 bs.$.y....U.s..@
0010 - cd 3f 0e 49 97 44 1a a8-39 4c 55 a8 2f 14 a8 98 .?.I.D..9LU./...
Start Time: 1745761853
Timeout : 7200 (sec)
Verify return code: 19 (self-signed certificate in certificate chain)
Extended master secret: no
Max Early Data: 0
---
read R BLOCK
test2.local:
❯ openssl s_client -connect localhost:8080 -servername test2.local
Connecting to 127.0.0.1
CONNECTED(00000003)
depth=2 CN=Nioca Root
verify error:num=19:self-signed certificate in certificate chain
verify return:1
depth=2 CN=Nioca Root
verify return:1
depth=1 CN=Nioca Intermediate
verify return:1
depth=0 CN=localhost, O=rerox test
verify return:1
---
Certificate chain
0 s:CN=localhost, O=rerox test
i:CN=Nioca Intermediate
a:PKEY: id-ecPublicKey, 384 (bit); sigalg: ecdsa-with-SHA384
v:NotBefore: Apr 27 13:16:56 2025 GMT; NotAfter: May 7 13:26:56 2026 GMT
1 s:CN=Nioca Intermediate
i:CN=Nioca Root
a:PKEY: id-ecPublicKey, 384 (bit); sigalg: ecdsa-with-SHA384
v:NotBefore: May 7 16:42:41 2024 GMT; NotAfter: Apr 24 18:52:41 2035 GMT
2 s:CN=Nioca Root
i:CN=Nioca Root
a:PKEY: id-ecPublicKey, 384 (bit); sigalg: ecdsa-with-SHA384
v:NotBefore: May 20 21:04:32 2023 GMT; NotAfter: Apr 19 18:52:32 2055 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIB/zCCAYWgAwIBAgIBAzAKBggqhkjOPQQDAzAdMRswGQYDVQQDDBJOaW9jYSBJ
bnRlcm1lZGlhdGUwHhcNMjUwNDI3MTMxNjU2WhcNMjYwNTA3MTMyNjU2WjApMRIw
EAYDVQQDDAlsb2NhbGhvc3QxEzARBgNVBAoMCnJlcm94IHRlc3QwdjAQBgcqhkjO
PQIBBgUrgQQAIgNiAASVcThGdDCuIpJtm/8F9n0fkSp8NRpnFn+NLYdJa6jIZdOV
C9xPFNJ7+Akw0irhPk+8ugenNbh5Lfbmpc6OE39zluG7GGCzTXj2QQ/3OTZEKZ+6
UFOv7pbgmA7prCq5FpWjgYwwgYkwHwYDVR0jBBgwFoAUK/pd61lzdkNyi8AcSE8P
ZWpnCzQwIQYDVR0RBBowGIIJbG9jYWxob3N0ggt0ZXN0Mi5sb2NhbDATBgNVHSUE
DDAKBggrBgEFBQcDATAdBgNVHQ4EFgQU5MFJ32uXU0FH+QcrFwdlI7usqigwDwYD
VR0TAQH/BAUwAwEBADAKBggqhkjOPQQDAwNoADBlAjEAotRKC3BmzT0oZ0uNfoRW
H4mUvVxqEgUriMVZ5sFcoOIXLAg2eUP1uWVntEKQX1MgAjBv5F4gu/LsMZWFHmfC
CltpnlLgMVG0wHZGhX9m0HN4KUmZIVtHHvMVK0JV40vhCso=
-----END CERTIFICATE-----
subject=CN=localhost, O=rerox test
issuer=CN=Nioca Intermediate
---
No client certificate CA names sent
Peer signing digest: SHA384
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1811 bytes and written 402 bytes
Verification error: self-signed certificate in certificate chain
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 384 bit
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 19 (self-signed certificate in certificate chain)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
Protocol : TLSv1.3
Cipher : TLS_AES_256_GCM_SHA384
Session-ID: 62FE61AAF41893EDF2A501BA494D5A73AB3996ADEFC8110E2A6D027FB3455B80
Session-ID-ctx:
Resumption PSK: 72AAE667644D396E2B6F1DC112569366843B41E6427C6B70CB8B4DFD518D2479E3D52AF277A78D05822586F2BB18C48B
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 86400 (seconds)
TLS session ticket:
0000 - 1e 31 b1 13 32 c8 c4 ed-54 4d 72 76 4b 52 ad 22 .1..2...TMrvKR."
0010 - 80 8a 6a e3 1e 9e 12 5f-bf 3e 23 d1 a3 32 34 3f ..j...._.>#..24?
Start Time: 1745761951
Timeout : 7200 (sec)
Verify return code: 19 (self-signed certificate in certificate chain)
Extended master secret: no
Max Early Data: 0
---
read R BLOCK
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
Protocol : TLSv1.3
Cipher : TLS_AES_256_GCM_SHA384
Session-ID: 2D924B1C9F038E26CC2D32DB3B4FA81B12FCEF1D65C859C5C0A5670F6A049AA4
Session-ID-ctx:
Resumption PSK: 40605E2D9A15ADDF4E88C86A2A7803B5CD28D8689E6BFFBAEBBDF88E57859075217AFF989C9D9AD4A57D21A71F4EA63B
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 86400 (seconds)
TLS session ticket:
0000 - 32 cb 26 53 69 8c 6b aa-e0 62 90 f7 0a 50 a4 80 2.&Si.k..b...P..
0010 - 15 f2 0f bd 9f de 03 50-de e5 c2 20 ae 9e 10 32 .......P... ...2
Start Time: 1745761951
Timeout : 7200 (sec)
Verify return code: 19 (self-signed certificate in certificate chain)
Extended master secret: no
Max Early Data: 0
---
read R BLOCK
And it fails of course for non-valid ones:
❯ openssl s_client -connect localhost:8080 -servername doesnotexist.local
Connecting to 127.0.0.1
CONNECTED(00000003)
C06203B7457F0000:error:0A000419:SSL routines:ssl3_read_bytes:tlsv1 alert access denied:ssl/record/rec_layer_s3.c:909:SSL alert number 49
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 329 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
Let me know what you think of it. I would gladly open a PR from the linked branch, as it is working nicely so far.
Thanks! That branch looks like a good pr candidate. I say open it up, and we will get to it when we can.
Hi @sebadob, Hi @johnhurt,
thank you for the PR #599 trying to solve the issue with SNI and rustls as described.
I also have interest in a solution that allows for certificate selection based on SNI but would at the same time like to take an approach that would allow for additionally updating the certs during the lifetime (e.g. short lived certs, ACME style updates, cert addition/deletion).
So far I've not looked into all the details but likely the API would be most flexible when allowing to set a Arc<dyn ResolvesServerCert> on the Listener or respectively Arc<dyn ResolvesClientCert> on the Connector. This approach would allow to implement solutions that e.g. are requested in #619.
https://docs.rs/rustls/latest/rustls/server/trait.ResolvesServerCert.html https://docs.rs/rustls/latest/rustls/client/trait.ResolvesClientCert.html
Useful Resolves*Cert implementations like a file based SNI resolver could be part of pingora-rustls.
Kind regards, Harald
Hi,
I've created a PR that allows to use a custom dyn ResolvesServerCert on the TlsListener.
As I've currently no usage for the dyn ResolvesClientCert as it is only related to mTLS client auth this is not part of the PR.
The PR is similar to the PR (#599) from @sebadob but directly exposes the required trait to build the Listener allowing to also handle other requirements.
A TLS SNI cert resolver can be configured with the exposed option.
Have a nice weekend. 😃
Kind regards, Harald
I was not aware of this trait in rustls. I like that idea of the dyn impl, provides even more flexibility.
Since nothing has been merged or reviewed yet, I may have another solution that could fix 3 issues at once.
I needed TLS hot-reloading for Rauthy and was not too happy with the existing crates because of internal locking or static initializers, so I created my own. While doing this, I created a dynamic certificate resolver, that can hot-reload rustls certificates at runtime. Just for the fun of it, I also added another wrapper that would load multiple of these dynamic hot-reloadable certificates and resolves based on SNI.
If I would now add another probably pretty simple function like reload_from_memory(), this could solve multiple issues at once.
Just for reference, what I can do for a single, auto-watched and hot-reloaded cert, is just
let ck = CertifiedKeyWatched::new(key_path, cert_path).await?;
rustls::ServerConfig::builder()
.with_no_client_auth()
.with_cert_resolver(ck)
I can do the same with another struct CertifiedKeysWatched which will take a Vec<BundleCert> just like in my PR.
Both of these impl ResolvesServerCert.
The file waching brings in notify as a dependency though, which is not too small, but it could be feature-gated.
Would you be interested in me doing an update to this PR? I currently plan on releasing the whole impl as a separate crate as well. Don't know if you would rather have it as a dependency or better a bit of copy & paste without external deps.
Edit:
Sorry forgot to mention: You can take a look at the current code here: https://github.com/sebadob/rauthy/tree/main/src/tls
The file waching brings in notify as a dependency though, which is not too small, but it could be feature-gated.
If the PR is close to ready and doesn't modify code outside of the rustls feature, then we could try to take a look - I do think that having a separate feature gate for this is the right call.
My original PR is ready since I opened it.
The other proposal with the automatic file-watching would require me to update it of course, but think it would be a very nice feature to have.
I updated my PR and added the dyn ResolvesServerCert as an additional option. My original idea of adding the whole "watched files + hot-reload" code into it felt a bit too high level and I was not sure about it. With the dynamic resolver, you could add it anyway, because I will release it as a separate crate. This is probably a bit cleaner when it comes to additional features.
If you would rather have this feature added to pingora at some point, I could do this in another PR.
For anyone interested, I released the watched hot-reload as a separate create: https://crates.io/crates/tls-hot-reload
This can be passed in directly into the custom resolver from my PR for instance.
For anyone interested, I released the watched hot-reload as a separate create: https://crates.io/crates/tls-hot-reload
This can be passed in directly into the custom resolver from my PR for instance.
pingora actually already have capability to hot-reload the TLS, you can take a look at openssl boringssl impl, it can provide in memory cert management with self-implemented ShardedLRU Cache for the Proxy and Gateway, combined with existing DynamicCert feature. which is very easy to integrated, you can take a look here at my code
