psf
requests fails to connect even if trusted leaf certificate is provided when server's cert chain is incomplete
Say you have a server that provides a cert chain
A -> B -> D
but D is actually signed by C, which is not provided by the chain.
(Unfortunately I have no control over this particular mistake so I can't get it fixed, and I don't even have a way to obtain C.)
Now obviously I can't just trust A and get things to work, but my problem is that even if I specify that I trust D verify="/path/to/d.pem, requests will still fail to connect.
This is in contrast with what curl does (--cacert /path/to/d.pem), where if the leaf is trusted it doesn't care that it can't find its issuer: it's specified as a root anyway.
Expected Result
Connects
Actual Result
requests.exceptions.SSLError: HTTPSConnectionPool(host='192.168.1.1', port=443): Max retries exceeded with url: /ws (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)')))
Reproduction Steps
Spawn an HTTPS server with an incomplete cert chain (e.g. no intermediate certificate). Make a request to it using requests.
System Information
$ python -m requests.help
{
"chardet": {
"version": null
},
"charset_normalizer": {
"version": "3.4.0"
},
"cryptography": {
"version": ""
},
"idna": {
"version": "3.10"
},
"implementation": {
"name": "CPython",
"version": "3.11.10"
},
"platform": {
"release": "6.12.33",
"system": "Linux"
},
"pyOpenSSL": {
"openssl_version": "",
"version": null
},
"requests": {
"version": "2.28.2"
},
"system_ssl": {
"version": "30300030"
},
"urllib3": {
"version": "1.26.20"
},
"using_charset_normalizer": true,
"using_pyopenssl": false
}
Hello,
I've taken a look at this issue as part of a class project and have identified the root cause. The issue stems from the hardcoded parameters used for the leaf and chain keys, which are too restrictive for certain use cases.
To address this, I've refactored the code to introduce more flexible parameters. This involved:
Adding a new classification for a single (leaf) key.
Implementing several conditional checks (if/then statements) to handle this new classification.
Ensuring the changes are robust by testing them extensively with pytest.
I've confirmed that these changes resolve the issue in my testing environment. I'm confident this solution addresses the limitations in the current key management logic.
Please let me know if you have any questions or would like me to provide a pull request. I'm happy to help.
To be clear @Ten0 have you been able to get leaf only verification working like this with just openssl s_client?
No, as far as I can tell s_client will give Verify return code: 21 (unable to verify the first certificate) at least with default parameters.
My thought was though, if curl allows this somehow maybe there's an openssl param that can be passed so that this is ignored if lower in the chain is trusted, but I have not investigated further.
https://docs.python.org/3/library/ssl.html#ssl.VERIFY_X509_PARTIAL_CHAIN is likely what needs to be configured here for requests like this but it isn't something we should be setting by default. The number of items in a trust store isn't a guarantee that the certificate is a leaf to be verified. And to determine an algorithm would require cryptography which we do not have a dependency on any longer to be able to check certain attributes which are not required for anything other than public CAs following baseline requirements from the CA/B Forum. Internal PKI may or may not follow those and thus complicate this.
This would be easier if we exposed an easy way to pass an SSL context but at the moment it requires a transport adapter that specifies one to urllib3 instead.
docs.python.org/3/library/ssl.html#ssl.VERIFY_X509_PARTIAL_CHAIN is likely what needs to be configured here
Looks like it indeed :)
The number of items in a trust store isn't a guarantee that the certificate is a leaf to be verified.
Definitely not, but it doesn't look like that option being set by default would imply a such behavior where it checks the number of items in the trust store? It seems instead to do the reasonable thing which is, just trust the intermediate cert as if it was a self-signed cert? (I'm not sure what you mean.)
but it isn't something we should be setting by default
As far as I'm concerned that seems pretty reasonable: I don't see what else one could one possibly mean by adding an intermediate cert to the trust store. That is indeed what curl does, and that opens more room for people to specify certs instead of having to disable certs verification (which is what I ended up having to do on the software that prompted me to open this issue).
@Ten0 I commented here but was referring to the PR created for this issue issue created by an LLM user where it checks for a single-certificate in the trust bundle.
It seems instead to do the reasonable thing which is, just trust the intermediate cert as if it was a self-signed cert? (I'm not sure what you mean.)
This is not how trust chains work. When a server and client are establishing a TLS connection, the server (to ensure that every client might work) must provide:
- it's leaf
- the issuer of that leaf
- any and all intermediates leading to the root
So if you have a architecture that looks like Root CA -> Intermediate -> Issuer -> Leaf Intermediate, Issuer, and Leaf must all be provided by the server. Web browsers can do something called AIA fetching to find missing links in the chain but most non-web-browser clients do not do that, and openssl does not give a good way to do this. That means that all that needs to be in the trust store is the Root, not an intermediate. An intermediate is signed by something else, and if we don't trust what signed it, we cannot trust it. That's why trust stores are also called Root Stores.
but it isn't something we should be setting by default
As far as I'm concerned that seems pretty reasonable:
Do you mean us not setting partial chain by default is reasonable?
I don't see what else one could one possibly mean by adding an intermediate cert to the trust store.
People do things like this without actually understanding why often times. Sometimes because they found it on a random website or because a statistical model tells them to based off some other website it was fed as part of its training data.
That is indeed what curl does, and that opens more room for people to specify certs instead of having to disable certs verification (which is what I ended up having to do on the software that prompted me to open this issue).
curl has specific flags to enable and control this behaviour. We do not have that kind of API surface area. The best way for us to enable this would be for us to accept an SSLContext or something that generates them. Thta isn't something we hvae a good way to add though, especially given our feature-freeze.
Disclaimer: Of course I am no particular authority on this, I'm just weighing in an opinion as a user with reasonable knowledge of PKIs and security in general, adding to that limited weight that of whoever is responsible for that kind of choice over at curl, to the extent that it can also be considered relevant here.
This is not how trust chains work
Thank you for your time on this explanation. I knew already though: my original message specifies as example a case where server's behavior constitutes a "mistake" that would require "fixing".
I mean that it sounds reasonable to me (and apparently to the curl people as well), security-wise, to have the same default behavior as curl, that is, if people explicitly ask to trust an intermediate, and the leaf has a valid chain leading up to that indermediate, we don't require to also trust the root and have a valid chain leading up to that root, because I do not see what else specifying an intermediate certificate as trusted could possibly mean, considering that as you said, root is otherwise both required and sufficient.
or because a statistical model tells them to
Even if they did trust an intermediate instead of the root when they trusting the root would have been more appropriate for their use-case, AFAICT that is not a security concern.
I don't see how that would cause security issues, and at least I know that it would fix mine, and I'm not saying this because an LLM told me so. 😅
In fact, I think that the least things are going to easily work, the more people who don't understand are going to just disable cert verification, which in turn looks like a more likely security issue than any one that would be caused by enabling this option by default.
It looks like this might also allow use-cases where one could trust less than the root, only a sub-portion of signatories, for a particular application, which could also possibly be more secure than being forced to trust the whole root which would then become a problem if any intermediate leaked.
In case of a chain CA -> Leaf cert where CA is unavailable next code works fine:
import requests
import ssl
import urllib3
class CustomSslAdapter(requests.adapters.HTTPAdapter):
def __init__(self, ssl_context=None, **kwargs):
self.ssl_context = ssl_context
super().__init__(**kwargs)
def init_poolmanager(self, connections, maxsize, block=False):
self.poolmanager = urllib3.poolmanager.PoolManager(
num_pools=connections,
maxsize=maxsize,
block=block,
ssl_context=self.ssl_context
)
my_leaf_certs = ['cert.crt']
context = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
context.verify_flags |= ssl.VERIFY_X509_PARTIAL_CHAIN
for leaf_cert in my_leaf_certs:
context.load_verify_locations(leaf_cert)
session = requests.Session()
session.mount('https://', CustomSslAdapter(ssl_context=context))
response = session.get('https://host:port')
print(response.status_code)
With deco implementation:
def verify_x509_partial_chain(my_leaf_certs, ssl_context=None):
def deco(get_session_func):
def deco2():
session = get_session_func()
context = ssl_context or ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
context.verify_flags |= ssl.VERIFY_X509_PARTIAL_CHAIN
print(context.verify_flags)
for leaf_cert in my_leaf_certs:
context.load_verify_locations(leaf_cert)
session.mount('https://', CustomSslAdapter(ssl_context=context))
return session
return deco2
return deco
@verify_x509_partial_chain(my_leaf_certs=['cert.crt'])
def get_session():
return requests.Session()
session = get_session()
response = session.get('https://host:port')
print(response.status_code)
With context manager implementation:
from contextlib import contextmanager
@contextmanager
def verify_x509_partial_chain(my_leaf_certs, ssl_context=None):
context = ssl_context or ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
context.verify_flags |= ssl.VERIFY_X509_PARTIAL_CHAIN
for leaf_cert in my_leaf_certs:
context.load_verify_locations(leaf_cert)
session = requests.Session()
session.mount('https://', CustomSslAdapter(ssl_context=context))
try:
yield session
finally:
pass
with verify_x509_partial_chain(my_leaf_certs=['cert.crt']) as session:
response = session.get('https://host:port')
print(response.status_code)
