Testing: Certificates incompatible with urllib3 2.4.0 and Python 3.13

[amotl]

Problem

urllib3 introduced a slightly breaking change for Python 3.13, which started tripping the test suite on nightly builds.

crate.client.exceptions.ConnectionError: Server not available, exception: 
HTTPSConnectionPool(host='localhost', port=65534): Max retries exceeded with url: / 
(Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] 
certificate verify failed: Missing Authority Key Identifier (_ssl.c:1028)')))

Details

@jvanasco explains the situation...

The default certificate does not have an AKID - which is considered to be RFC non-compliant. It was generated before support was introduced in 2019. This breaks against the default Python 3.13 ssl context with urllib3 2.4.0 now, which aligned their ssl_context with cPython. It was released yesterday.

Solution

.... and also provides a solution:

The current version of minica generates certs that are compliant, so please regenerate [testing certificates] with current minica.

Thanks!

References

• https://github.com/urllib3/urllib3/issues/3571
• https://urllib3.readthedocs.io/en/stable/changelog.html

[amotl]

This comment by @bdraco (thanks!) also seems relevant in this context:

Thats likely due to the openssl version requiring the certificate to meet newest CAB forum requirements and comply with the RFC https://www.rfc-editor.org/rfc/rfc5280#section-4.2.1.9

• https://github.com/aio-libs/aiohttp/issues/9869#issuecomment-2504502552

[amotl]

For people using aiohttp, the solution to configure a more relaxed SSL client context as provided by @szelenka-cisco (thanks!) is included in the same discussion:

• https://github.com/aio-libs/aiohttp/issues/9869#issuecomment-2688043767

It seems to be revolving around disabling FIPS mode?

[amotl]

Reading about matters a bit closer [^1], I think CA certificates MUST adhere to stronger requirements about their certificates now, like providing Authority Key Identifier and Subject Key Identifier elements.

Clients are starting to validate those requirements now, as the error message Missing Authority Key Identifier conveys.

[^1]: ... but still too quick that my assessments can also be wrong. So, please correct me where I am wrong.

[jvanasco]

Since I got some messages after being cross-referenced to this, hopefully I can help a bit. To clarify:

• Python introduced some changes to the basic validation in which, amongst other things, AKID became required.
• urllib3 aligned their behavior starting in the 2.4 release for py3.13 and up

IMHO the easiest solutions right now are to either:

1- pin urllib3<2.4 which will let things just work as-is. 2- regenerate your cert. the new version of minica is compatible. i generated one here if you want to use it for your tests: https://github.com/aptise/peter_sslers/tree/main/tests/test_configuration/pebble/test-alt/certs

(I discovered the issue so quickly, because CI tests against the default pebble install failed, but ones against my alternate passed)

The other ticket cross-referenced is from a tangential issue - verification failed due to a different alignment within the SSL context. The one you're getting hit with is the new requirement of VERIFY_X509_STRICT (though VERIFY_X509_PARTIAL_CHAIN also changed).

You can also reimplement how the SSL context is created via create_urllib3_context. The relevant code for the py3.13 change is here: https://github.com/urllib3/urllib3/blob/main/src/urllib3/util/ssl_.py#L333-L340

You could just generate that default context, catch py313 to undo the changes, and pass that context explicitly. I found version pinning to be easiest.

IMHO the best solution would be updating the cert and adapting code to enforce the py313 changes on all platforms. That latter bit is a pain. I should file a request against urrlib3 over this.

[amotl]

Dear @jvanasco. Thank you so much for sharing your excellent elaborations and suggestions.

For the sake of completeness: We have been tripped by this beyond the Python driver for the CrateDB database, but also with crate-operator on Python 3.13, that's why I relayed information about how to configure the SSL context for aiohttp: In this case, SSL certificates employed by the AKS K8s HTTP API don't use AKID attributes yet. As we can't do anything about it, we need to apply one of the workarounds as outlined in your suggestions.

We summarized them like this:

This means either do not use Python 3.13 at all, adjust its default SSL context to be more relaxed, for example before making connections using its built-in urllib library, or, in many other cases, just downgrade to urllib3<2.4 until relevant CA certificates have been replaced by modern ones.

We may update this statement to also include advises about aiohttp and httpx, accompanied by further details from your pen about adjusting Python's vanilla SSL context. 🌻

Within this repository, we will use minica to re-generate testing certificates, as you suggested. Until then, we added a build-time constraint d32b74a55a to fix the build. We don't expect too many woes on runtime aspects, as we think most of relevant public CA certificates have been upgraded already, so we guess this issue is mostly hitting people who are still using old self-signed certificates.

[amotl]

IMHO the best solution would be updating the cert and adapting code to enforce the py313 changes on all platforms. That latter bit is a pain. I should file a request against urllib3 over this.

I think it is a Python-wide consensus to enforce stronger constraints starting with Python 3.13 only. I am inclined to believe there have been plenty of discussions around this very detail, and I don't disagree too much with that decision.

[jvanasco]

In this case, SSL certificates employed by the AKS K8s HTTP API don't use AKID attributes yet.

Oh gosh, that's messy.

Your users are probably getting hit by ssl.create_default_context which is used by the stdlib's urllib and asyncio - and where those urllib3 changes were backported from::

• docs: https://docs.python.org/3/library/ssl.html#ssl.create_default_context
• source: https://github.com/python/cpython/blob/14305a83d381ccdcae814abd9e2c28dda066720b/Lib/ssl.py#L682

AFAIK, in those situations you would need to either pin python under 3.13 OR explicitly create and pass-in a context: ssl.SSLContext var to the various request functions in urllib and asyncio

I think it is a Python-wide consensus to enforce stronger constraints starting with Python 3.13 only. I am inclined to believe there have been plenty of discussions around this very detail, and I don't disagree too much with that decision.

I agree with you here. There is a slight nuance that may not have been conveyed in my above comment - I intended to address the lack of an easy way to explicitly opt-in earlier Python versions to the new stronger constraints, which was not part of the discussion on this urllib3 change. I just offered to write a PR to accomplish this: the stricter behavior would apply to py>=3.13 OR the presence of an environment variable.

[amotl]

Thanks. Yeah, with CrateDB Operator talking to AKS, we will stick with using Python 3.12 for now. Others that use urllib3, we will just downgrade to urllib3<2.4.

On another detail when sampling other popular HTTP libraries for Python, just looking for "Missing Authority Key Identifier" on the httpx repository, we discovered https://github.com/encode/httpx/pull/3434, where cryptography's changelog for version 44.0.0 - 2024-11-27 says:

Relax the Authority Key Identifier requirements on root CA certificates during X.509 verification to allow fields permitted by RFC 5280 but forbidden by the CA/Browser BRs.

So it looks like httpx and possibly other libraries that use the cryptography package yield yet another kind of behaviour according to its default configuration in this regard, which seems to take a more relaxed approach, regardless of Python versions?

[amotl]

Regarding the vanilla Python SSL context, because you referred to ssl.create_default_context(), do you think this configuration, strictly using ssl.CERT_REQUIRED, would be the recommended way to take a more relaxed approach which skips the newer defaults ssl.VERIFY_X509_PARTIAL_CHAIN and ssl.VERIFY_X509_STRICT altogether?

ctx = ssl.create_default_context()
ctx.verify_flags = ssl.CERT_REQUIRED

[jvanasco]

I am not familiar enough with the current flags to comment on that.

I would probably just remove the two new flags, with something like...

ctx = ssl.create_default_context()
if sys.version_info >= (3, 13):
    # Added in version 3.4
    ctx.verify_flags = ctx.verify_flags & ~ssl.VERIFY_X509_STRICT
    # Added in version 3.10.
    ctx.verify_flags = ctx.verify_flags & ~ssl.VERIFY_X509_PARTIAL_CHAIN

Background on the change is here:

• Discussion https://discuss.python.org/t/ssl-changing-the-default-sslcontext-verify-flags/30230
• Issue https://github.com/python/cpython/issues/107361
• PR https://github.com/python/cpython/pull/112389

In terms of the 5280 stuff::

Of relevance, the py>=3.13 security policy enforces the requirement of AKIDs.

The "relax" PR in cryptography just allows for additional fields the CA/B Forum prohibits, but the RFC permits and have been "out in the wild" across trusted root programs for a while.

The RFC 5280 stuff is tied to this PR, and basically the inverse of our shared problem :: https://github.com/pyca/cryptography/pull/11462/files

[amotl]

Hi. We also used minica now to generate new certificates. Finally!

• GH-754

@jvanasco: Thanks a stack for your guidance and recommendations.

[jvanasco]

Glad to help!

I’m traveling now, but FYI on a related issue you may run into — if you are supporting multiple Python versions, please be aware that you may need to reimplement the urllib3 create_xxx_context function. I forgot the exact name and my connection is spotty now. That function has logic that only applies the new defaults to py313 and above - there is no way to opt-in lower versions to the new flags, and the maintainers are not interested in supporting such a feature through the lifetime of 3.12.

I have our projects re-implementing this function but for all Python versions, with notes to remove it in favor of the original once 3.12 is EOL.

Best, jv

[amotl]

Thank you Jonathan. Enjoy your travels, and merry christmas.

Please help me understanding your recommendation: Are you referring to the topic how to make it possible to provide a variant of the library for all Python versions, which can talk to https endpoints that don't provide certificates including AKIDs yet, because there certainly might be some of those systems in the wild, and how to make urllib3 comply to that?

If this is the case, I've also thought about this. Currently, I'm inclined to let this run through a scream test, but I am also open to be convinced into a different direction, by providing a custom SSL context function to make this option configurable from the start, like you might be suggesting?

/cc @surister, @seut

[jvanasco]

Thanks!  Just a quick pre-Holiday trip (so we can relax next week!). Hope your winter plans are festive.The issue is much simpler and more about api/behavior compatibility than “loosening” the issues.  The current behavior actually does what you hope for - on py3.12 and lower, it will NOT apply the default flags for py3.13.

The problem (to me) is that one can’t “cleanly” configure a py3.12 to behave as py3.13 without passing explicit kwargs for whichever behaviors they want - as the code is tied to a ‘sys.version()’ executed in the function call.  So calling create_ssl_context will behave differently depending on version.  I generally avoid an explicit configuration on this, so I can rely on security upgrades (ironically, like this one!) So you’re possibly in a situation where continuing support for py3.12 will mean you have to do one of the following:

• maintain test code to expect these differences (I opted for that)
• maintain a compatibility layer to standardize onto 3.12 or 3.13

IMHO there is no wrong choice here - given this popping up just before the holidays, I just wanted to drop this on your radar as something to plan around or beware of, in case you suddenly see unit tests passing in one environment and failing in others.  A few friends went through debugging this same issue too. It’s the last thing anyone needs before heading on a break :)

There is a (closed) issue here:

- https://github.com/urllib3/urllib3/issues/3605#issuecomment-3415451890

[amotl]

Thank you very much. I hear you, and if others also agree, it will be better to release only after the winter breaks. On the other hand, we don't intend to pin the version of urllib3 to any specific release anyway, so I guess users will always have the freedom of choice by downgrading to previous library behaviours if they want to, in case something breaks?

[amotl]

As far as I am concerned, now after the patch being merged,

• GH-754

I think we can also close the ticket. However, I wanted to reassure myself that we don't miss any recommendations by @jvanasco (thank you again!). wdyt, @surister, @seut?