dotnet
SslStream only sends chain certs if they're in the Windows Cert Store
I'm using the RabbitMQ C# Client, which under the hood uses SslStream. I'm having an issue where clients are unable to authenticate using x509 certificates if intermediate certificates are involved - such a chain looks like:
Root CA -> Issuing CA -> Issued Client Certificate
Using Wireshark I can see that when authenticating as a client, SslStream is sending only the leaf certificate, which is causing a certificate handshake error. However, if the Root CA and Issuing CA are added to the Windows Certificate Store as trusted roots, then SslStream sends all 3 certificates, and RabbitMQ is happy.
The certificate I'm using as the client cert is a PKCS#12 file that contains the whole chain (as X509Certificate2). So, the question is if there is any way to force SslStream to send the whole chain when authenticating as a client, even if the chain certs are not in the Windows Certificate Store?
...except you're using the file, not a cert from the store, as the client id, right? Very probably it's not loading the entire chain from the file (it only actually loads one), so the leaf has a reference value to the issuer cert (thumbprint? not sure), but no idea what the actual cert is.
If I remember right, you have to load the entire set of certs into a runtime store, then grab the leaf certificate from that. The reason it works if it's in the windows cert store is that it does do lookup there by default.
@Clockwork-Muse I'm currently selecting the certificate from an X509CertificateCollection that contains the full chain.
I figured the solution might involve creating some kind of temporary X509Store, but if I create a unique one for my app and select a certificate from it, it's still the case that only the leaf is sent - it only seems to send the whole chain if the others certs are in the My store (strangely, not the Machine store).
If you have any info on how to create a temporary store only visible to the app and have SslStream send the whole chain, some guidance would be much appreciated.
I've checked from Linux, and observed the same behavior - the chain of intermediate certs is sent if I use the My certificate store, but not if I create a unique one with new X509Store("whatever").
I guess this is happening in SecureChannel, but that is a bit of a whale, and I'm struggling to find where in the code this actually takes please?
I've managed to get it working as expected on Linux, by using the SSL_CERT_FILE environment variable:
SSL_CERT_FILE=/opt/my-app/etc/ca-bundle.crt ./My.App
When doing this, SslStream is sending the intermediate certificate as well as the leaf, and RabbitMQ is happy.
Is there anything like this for Windows? @bartonjs, I guess if anyone would know it would be you? ;)
@cocowalla - well, often, SSL_CERT_FILE/SSL_CERT_DIR are set to a global location, so it ends up being the equivalent of the Windows certificate store (I'm currently cursing Python, because it uses about 3/4 different sets of certs during setup pulled in from different packages). I guess the closest equivalent would be certs that only certain users would have permission to access (which you'd normally want to do for the private key anyways).
I thought I'd discovered a way to load/send the entire chain when dealing with a different issue, but I can't recreate it now, possibly I just imagined it.
What's your usecase here, that you wouldn't be adding the certs to the store anyways?
Regarding SSL_CERT_FILE/SSL_CERT_DIR, these certs are only for use by this particular application, so the env var is set just for the application.
My usecase was a (failed!) attempt to keep things simple by using the same approach on both Windows and Linux (keys stored in the filesystem, protected by ACLs). I've decided to give up; I'll use SSL_CERT_FILE on Linux, and the My root and intermediate stores on Windows.
I do however still think that SslStream should have some means of providing chain certs from ephemeral stores or a X509Certificate2Collection.
While I agree that it would be nice if SslStream would send the entire cert (which would probably require the actual cert domain types keeping their reference, or whatever), there's a problem with that: You're sidestepping the normal mechanism for chain resolution on the platforms. If the cert end up in a resource bundle or something, it ends up being more difficult for the end user to configure.
I don't really see it as side-stepping anything; more augmenting the existing mechanism. The My store would still be used, but the ephemeral store or X509Certificate2Collection the cert came from would additionally be used to locate chain certs. I imagine most devs would expect this behaviour; if you load a chain from a PKCS12 file, it seems counter-intuitive to only use the leaf cert and completely ignore the chain certs.
For your last point, I'd argue that most users would be comfortable replacing a file, and probably haven't even heard of the Windows Certificate Store ;)
I don't really know the SChannel APIs (which provide TLS for us on Windows), but I think that they only take the single certificate, then internally do the chain building for sending the intermediates.
The Linux and macOS TLS APIs are a little more raw, so they might be more amenable to such a feature, but I wouldn't add a new feature that doesn't work on Windows (71% of the usage of .NET Core, as of last summer).
@bartonjs ah, that explains why I couldn't find anything in SecureChannel that sends the intermediate certs - I didn't realise it was using an SChannel API that's hard-coded to check against specific stores.
I agree it makes no sense to add this unless it works across all platforms.
I like to bring this issue back to the table.
We have a aspnet core based service that should be deployed onto clients windows computers. They are most likely not publicly available on the internet, and the http API that our service provides should be secured. We have a self-signed root CA and an intermediate CA and this issues a certificate for each installation. This way we can also do client-certificate auth with our service as the client.
We really want to avoid to install our root and intermediate certificates into the trusted root CA store of the clients computers. We deem this a bad security practice and really want to avoid it.
Technically it is sufficient to use certificate pinning to our root CA cert within our dedicated client application. There is no need that any computer should be needed to completely trust our cert.
So there needs to be a way to tell kestrel on windows (which uses SslStream) to not only send the actual server certificate we configured but the complete chain, without installing the chain in the trusted root store of the computer.
This is pretty counter intuitive if you create a .pfx file containing the complete chain (the server cert, the servers private key as well as both the root and intermediate CA certs), tell kestrel to use this file, and then, when its not working, find out that the X509Certificate2 only represents a single cert from the file and not the chain, and there is no other way to tell the system where to find the certificates that it should send along except for exposing the computer of your client to a risk that usually should be prevented at all costs.
So, what is the actual plan to overcome this issue? Is there anything we could do to help get this sorted out?
So, what is the actual plan to overcome this issue? Is there anything we could do to help get this sorted out?
I agree with your thoughts about the importance of this scenario. It should "just work".
As far as making progress on this issue, we need to understand what the capabilities/API of the underlying TLS stacks (SCHANNEL for Windows, OpenSsl for Linux, MacOS) are. At this point, we aren't sure the platforms support this functionality.
I have almost exactly the same situation as @gingters . We would like to send the full chain if possible - and would also like to avoid placing root certificates in a user/computer-wide store for a variety of reasons. As mentioned, telling Kestrel to use a .pfx which provides a full chain and then having them conceptually "dropped on the floor" is not a good experience, and recent efforts (#31944) to be able to read formats that are often associated with carrying the full chain highlights this.
(For anyone in the same situation who needs to have clients validate a certificate presented without its chain: if you can prime a list of potential certificates and get a grip on their chains beforehand, that's a good workaround. We can't, because the set of potential certificates is large and volatile. Our workaround right now is for the client to connect, receive the certificate, disconnect, use a secondary server or endpoint to retrieve the full chain and then connect again with the answer in hand. The alternative is to stall synchronously in the certificate validator, which is not only an incredibly bad idea but can also look hostile and malicious to the server, and is liable to be tripped up by defenses or timeouts in Kestrel or Schannel. Asynchronous certificate validation might have fixed this if this step wasn't supposed not to have huge pauses in it in the first place.)
Schannel not supporting this is a reasonable explanation why, but considering alternate stacks can provide a solution it would be good to be able to opt into them. (And yes - I realize this is a big hammer, but it would also be able to pound in many nails.)
Hi. We are having the exact same problem. Certificate trust chain is broken because only the server leaf certificate is sent without the intermediate CA (and root CA). The PFX file had all the certs bundled. Very unexpected and problematic behavior.
As a workaround I am using HAProxy to terminate the SSL traffic and forward it to unencrypted port.
the server part will be fixed in 5.0. There is now option to pass CertificateContext with all chain - perhaps loaded from pfx or PEM file. However, on Windows that will add intermediates to the store if needed - there is no other way how to deal with it as the handshake actually does not happen in the same process space. That was recommended by Windows platform developers.
The client part will need some more work and thinking.
It also seems like this morphed from client to server side back in 2019. Since this is probably too late for @cocowalla, I'm thinking about closing this and perhaps moving the server discussion to separate issue if needed - and the 5.0 behavior does not seem sufficient - #35844.
Good to see some progress and that this issue is given some thought and attention. (Less good that the workaround is all that can happen on Windows, but that's not the fault of the .NET team.)
Reassigning to System.Net.Security since it's about SslStream. The original issue was about client certs, and so far the SslCertificateContext type (which fixed it for the server role) isn't available to the client role.
Tagging subscribers to this area: @dotnet/ncl, @vcsjones See info in area-owners.md if you want to be subscribed.
Issue Details
I'm using the RabbitMQ C# Client, which under the hood uses SslStream. I'm having an issue where clients are unable to authenticate using x509 certificates if intermediate certificates are involved - such a chain looks like:
Root CA -> Issuing CA -> Issued Client Certificate
Using Wireshark I can see that when authenticating as a client, SslStream is sending only the leaf certificate, which is causing a certificate handshake error. However, if the Root CA and Issuing CA are added to the Windows Certificate Store as trusted roots, then SslStream sends all 3 certificates, and RabbitMQ is happy.
The certificate I'm using as the client cert is a PKCS#12 file that contains the whole chain (as X509Certificate2). So, the question is if there is any way to force SslStream to send the whole chain when authenticating as a client, even if the chain certs are not in the Windows Certificate Store?
| Author: | cocowalla |
|---|---|
| Assignees: | - |
| Labels: |
|
| Milestone: | Future |
I think I am having the same issue here and so far (after days of trying) no solution for that.
Our backend (which we can not touch) requires that two intermediate certificates are send along with the client certificate. In Wireshark we can observe, that only the client certificate is sent. When trying this with e.g. Python Requests or CURL this does not cause any issues.
We are now thinking about switching technology since we can not come up with a proper solution for this. Does anyone have a workaround for this problem? It is a bit shocking, that something "simple" as:
curl -v -s -k\ --request GET\ --key-type PEM --key /PATH_TO_CERT_KEY --cert-type PEM --cert /PATH_TO_CERT_WITH_CHAIN https://SOMEHOSTNAME.COM
seems to be impossible to achieve with .Net Core...
This is the structure of our certificate:
-----BEGIN CERTIFICATE----- // Client cert -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- // intermediate -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- //intermediate -----END CERTIFICATE-----
Does anyone have a workaround for this problem?
The workaround is to install the client intermediate certificates into the Windows certificate store (I think the "My" store is the right place).
I think the "My" store is the right place
Intermediates should go in StoreName.CertificateAuthority, which the Windows Certificate Store UI calls "Intermediate Certificate Authorities". (They'll probably work in the My store, but that's not the expected place)
Well, I noticed, that I can access my (mac OS) store using a X509Store. However, this is only my dev machine and we have to deploy this stuff to Microsoft Azure on Linux driven function apps. Did not test this yet, but might give it a try, since @cocowalla wrote:
I've checked from Linux, and observed the same behavior - the chain of intermediate certs is sent if I use the My certificate store, but not if I create a unique one with new X509Store("whatever").
So I was thinking to maybe add and remove the intermediate certificates on the fly when executing our tests (context: we have hundreds of virtual entities, that are alle equipped with client certificates. Therefore it would be cool not to "pollute" the systems too much by adding certificates to the store).
Thank you @bartonjs for the hint with the CertificateAuthority, I will have a look into this tomorrow!
This will probably be solved by #71194 in 8.0. Note that on Windows there is no API to pass specific certificates. SslStreamCertificateContext puts the certificates to stores automatically as needed so callers do not need to worry about it. This is primarily about intermediates. The certificate AND key still needs to be in a store on Windows as the handshake happens in separate process. (#23749)
Hi @bartonjs @rzikm I wrote the following code and tried to send it out. Should this be working? I still don't get the certificates sent with the request (no matter if I use StoreName.CertificateAuthority or StoreName.My). From what I understand, the SslStreamCertificateContext should now grab the certificates implicitly from the store when sending out the request:
private void TestRegistration(X509Certificate2 clientCert, X509Certificate2 l1, X509Certificate2 l2, X509Certificate2 root, string payload)
{
if (clientCert.HasPrivateKey == false)
throw new Exception("Client certificate is missing private key");
// Prime Root store
var storeRoot = new X509Store(StoreName.AuthRoot);
storeRoot.Open(OpenFlags.ReadWrite);
storeRoot.Add(root);
storeRoot.Close();
// Prime CA store
var storeAuthorities = new X509Store(StoreName.CertificateAuthority);
storeAuthorities.Open(OpenFlags.ReadWrite);
storeAuthorities.Add(l1);
storeAuthorities.Add(l2);
storeAuthorities.Close();
var handler = new HttpClientHandler();
handler.ClientCertificateOptions = ClientCertificateOption.Manual;
handler.SslProtocols = SslProtocols.Tls12;
// Prime certificate store
var storeCerts = new X509Store(StoreName.My);
storeCerts.Open(OpenFlags.ReadWrite);
storeCerts.Add(clientCert);
// Freshly fetch the client cert from store
var storedCertificate = storeCerts.Certificates.Where(x => x.SerialNumber == clientCert.SerialNumber).First();
if (storedCertificate.HasPrivateKey == false)
throw new Exception("Client certificate is missing private key");
handler.ClientCertificates.Add(storedCertificate);
storeCerts.Close();
handler.ServerCertificateCustomValidationCallback =
(httpRequestMessage, cert, cetChain, policyErrors) =>
{
return true;
};
var content = new StringContent(payload, System.Text.Encoding.UTF8, "application/json");
var client = new HttpClient(handler);
var result = client.PutAsync("https:/MY_BACKEND_HOSTNAME.com/api/register",
content).GetAwaiter().GetResult();
if (result.StatusCode != HttpStatusCode.OK)
{
var statuscode = result.StatusCode;
throw new SuccessException("Registration failed with " + statuscode.ToString());
}
}
store.Add(clientCert);
I do not know if this is the error, but you are adding the client certificate to the Certificate Authority store too. You may need to add it to the .My store. You may also need to Close/Dispose a store for the changes to take effect - I don't see anything confirming or denying this in the documentation.
Hi @JesperTreetop , I updated the code above ^ , to now have two stores that are closed before sending the request. Still not working. Is is pretty hard to guess what is going on in the back when doing it like this. I was wondering how the client will know, that the intermediates belong to the client certificate added to handler.ClientCertificates.Add(storedCertificate)
I was wondering how the client will know, that the intermediates belong to the client certificate
In the standard X509 certificate chain-of-trust way. The client certificate is signed/issued by the intermediate certificate and the intermediate certificate is signed/issued by the root CA certificate. Each certificate also contains "issuer" metadata through which you can find an issuing certificate. See "Issuer" here, for example.
Come to think of it, per the documentation, the intermediate certificate should go in the StoreName.CertificateAuthority store and the root CA certificate should go in the StoreName.Root store. The intermediate certificate would be unable to find the root CA certificate (which issued the intermediate certificate) if only looking in the root store if it hadn't been added to the root store.
Hi @JesperTreetop, thank you for looking into this. I have now (again) updated the code above ^ and put the 3rd party root certificate into the StoreName.AuthRoot. Unfortunately this also did not help. In case you wonder, this is the certificate structure:
root
|_ intermediate L1
|_ intermediate L2
|_ client certificate
They form a valid chain, which I also checked using the openssl verify command. PS: I know that is odd, that the client needs to send the L2 and L1 certs to the server, but unfortunately this is out of my hands and we simply must do it this way :/
I think maybe this is better answered by someone involved with that code. I'm just some dude guessing. It would be great if this just worked as intended out of the box.