kubernetes-client
urllib3 v2.4.0 on Python 3.13 doesn't work with EKS
What happened (please include outputs or screenshots):
The following exception is raised whenever calling the Kubernetes API of an EKS cluster:
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='0123456789deadbeef01234567890000.gr7.us-east-1.eks.amazonaws.com', port=443): Max retries exceeded with url: /version/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Missing Authority Key Identifier (_ssl.c:1028)')))
Full stacktrace
Traceback (most recent call last):
File "/path/to/venv/lib/python3.13/site-packages/urllib3/connectionpool.py", line 464, in _make_request
self._validate_conn(conn)
~~~~~~~~~~~~~~~~~~~^^^^^^
File "/path/to/venv/lib/python3.13/site-packages/urllib3/connectionpool.py", line 1093, in _validate_conn
conn.connect()
~~~~~~~~~~~~^^
File "/path/to/venv/lib/python3.13/site-packages/urllib3/connection.py", line 741, in connect
sock_and_verified = _ssl_wrap_socket_and_match_hostname(
sock=sock,
...<14 lines>...
assert_fingerprint=self.assert_fingerprint,
)
File "/path/to/venv/lib/python3.13/site-packages/urllib3/connection.py", line 920, in _ssl_wrap_socket_and_match_hostname
ssl_sock = ssl_wrap_socket(
sock=sock,
...<8 lines>...
tls_in_tls=tls_in_tls,
)
File "/path/to/venv/lib/python3.13/site-packages/urllib3/util/ssl_.py", line 480, in ssl_wrap_socket
ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
File "/path/to/venv/lib/python3.13/site-packages/urllib3/util/ssl_.py", line 524, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/mk9waz7zbq8hxm6sxhwn88hhcwmgsa84-python3-3.13.3/lib/python3.13/ssl.py", line 455, in wrap_socket
return self.sslsocket_class._create(
~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
sock=sock,
^^^^^^^^^^
...<5 lines>...
session=session
^^^^^^^^^^^^^^^
)
^
File "/nix/store/mk9waz7zbq8hxm6sxhwn88hhcwmgsa84-python3-3.13.3/lib/python3.13/ssl.py", line 1076, in _create
self.do_handshake()
~~~~~~~~~~~~~~~~~^^
File "/nix/store/mk9waz7zbq8hxm6sxhwn88hhcwmgsa84-python3-3.13.3/lib/python3.13/ssl.py", line 1372, in do_handshake
self._sslobj.do_handshake()
~~~~~~~~~~~~~~~~~~~~~~~~~^^
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Missing Authority Key Identifier (_ssl.c:1028)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/path/to/venv/lib/python3.13/site-packages/urllib3/connectionpool.py", line 787, in urlopen
response = self._make_request(
conn,
...<10 lines>...
**response_kw,
)
File "/path/to/venv/lib/python3.13/site-packages/urllib3/connectionpool.py", line 488, in _make_request
raise new_e
urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Missing Authority Key Identifier (_ssl.c:1028)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "<python-input-3>", line 1, in <module>
client.get_code()
~~~~~~~~~~~~~~~^^
File "/path/to/venv/lib/python3.13/site-packages/kubernetes/client/api/version_api.py", line 61, in get_code
return self.get_code_with_http_info(**kwargs) # noqa: E501
~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
File "/path/to/venv/lib/python3.13/site-packages/kubernetes/client/api/version_api.py", line 128, in get_code_with_http_info
return self.api_client.call_api(
~~~~~~~~~~~~~~~~~~~~~~~~^
'/version/', 'GET',
^^^^^^^^^^^^^^^^^^^
...<11 lines>...
_request_timeout=local_var_params.get('_request_timeout'),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
collection_formats=collection_formats)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/path/to/venv/lib/python3.13/site-packages/kubernetes/client/api_client.py", line 348, in call_api
return self.__call_api(resource_path, method,
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
path_params, query_params, header_params,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...<2 lines>...
_return_http_data_only, collection_formats,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_preload_content, _request_timeout, _host)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/path/to/venv/lib/python3.13/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
response_data = self.request(
method, url, query_params=query_params, headers=header_params,
post_params=post_params, body=body,
_preload_content=_preload_content,
_request_timeout=_request_timeout)
File "/path/to/venv/lib/python3.13/site-packages/kubernetes/client/api_client.py", line 373, in request
return self.rest_client.GET(url,
~~~~~~~~~~~~~~~~~~~~^^^^^
query_params=query_params,
^^^^^^^^^^^^^^^^^^^^^^^^^^
_preload_content=_preload_content,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_request_timeout=_request_timeout,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
headers=headers)
^^^^^^^^^^^^^^^^
File "/path/to/venv/lib/python3.13/site-packages/kubernetes/client/rest.py", line 244, in GET
return self.request("GET", url,
~~~~~~~~~~~~^^^^^^^^^^^^
headers=headers,
^^^^^^^^^^^^^^^^
_preload_content=_preload_content,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_request_timeout=_request_timeout,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
query_params=query_params)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/path/to/venv/lib/python3.13/site-packages/kubernetes/client/rest.py", line 217, in request
r = self.pool_manager.request(method, url,
fields=query_params,
preload_content=_preload_content,
timeout=timeout,
headers=headers)
File "/path/to/venv/lib/python3.13/site-packages/urllib3/_request_methods.py", line 135, in request
return self.request_encode_url(
~~~~~~~~~~~~~~~~~~~~~~~^
method,
^^^^^^^
...<3 lines>...
**urlopen_kw,
^^^^^^^^^^^^^
)
^
File "/path/to/venv/lib/python3.13/site-packages/urllib3/_request_methods.py", line 182, in request_encode_url
return self.urlopen(method, url, **extra_kw)
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
File "/path/to/venv/lib/python3.13/site-packages/urllib3/poolmanager.py", line 443, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
File "/path/to/venv/lib/python3.13/site-packages/urllib3/connectionpool.py", line 871, in urlopen
return self.urlopen(
~~~~~~~~~~~~^
method,
^^^^^^^
...<13 lines>...
**response_kw,
^^^^^^^^^^^^^^
)
^
File "/path/to/venv/lib/python3.13/site-packages/urllib3/connectionpool.py", line 871, in urlopen
return self.urlopen(
~~~~~~~~~~~~^
method,
^^^^^^^
...<13 lines>...
**response_kw,
^^^^^^^^^^^^^^
)
^
File "/path/to/venv/lib/python3.13/site-packages/urllib3/connectionpool.py", line 871, in urlopen
return self.urlopen(
~~~~~~~~~~~~^
method,
^^^^^^^
...<13 lines>...
**response_kw,
^^^^^^^^^^^^^^
)
^
File "/path/to/venv/lib/python3.13/site-packages/urllib3/connectionpool.py", line 841, in urlopen
retries = retries.increment(
method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2]
)
File "/path/to/venv/lib/python3.13/site-packages/urllib3/util/retry.py", line 519, in increment
raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='0123456789deadbeef01234567890000.gr7.us-east-1.eks.amazonaws.com', port=443): Max retries exceeded with url: /version/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Missing Authority Key Identifier (_ssl.c:1028)')))
What you expected to happen:
The exception shouldn't be raised and the call to the Kubernetes API should be made successfully.
How to reproduce it (as minimally and precisely as possible):
Use the latest version of this project with urllib3 v2.4.0 on Python 3.13.
import kubernetes
kubernetes.config.load_config() # Should load a config for an EKS cluster
client = kubernetes.client.VersionApi()
client.get_code()
Anything else we need to know?:
This seems to be caused by the following change in urllib3 v2.4.0: issue, PR, which only takes effect on Python 3.13.
I've only experienced the issue with EKS, which must use self-signed certificates that aren't fully compatible with RFC 5280, notably because they don't provide an Authority Key Identifier.
I don't know if the same issue is the case of other Kubernetes providers.
Environment:
- Kubernetes version (
kubectl version): v1.32.3-eks-bcf3d70 - OS (e.g., MacOS 10.13.6): macOS 15.4.1
- Python version (
python --version): 3.13.3 - Python client version (
pip list | grep kubernetes): 32.0.1
We've run into this issue as well. AWS support told us this is only the case with EKS clusters created with version 1.16 and below. Creating a new cluster would fix the issue, but updating does not.
As a workaround:
client_configuration = None
if sys.version_info >= (3, 13):
# https://docs.python.org/3/whatsnew/3.13.html#ssl
client_configuration = Configuration()
client_configuration.verify_ssl = False
api_client = ApiClient(configuration=client_configuration)
kubernetes.client.VersionApi(api_client)
Of course, it's insecure to completely disable SSL verification, but it does avoid the issue.
We ran into this as well, with an on-prem RKE 1.28 cluster.
The issue seems to be related to a change in behavior in the urllib3 library starting with 2.4.0: https://github.com/urllib3/urllib3/releases/tag/2.4.0
Added
verify_flagsoption tocreate_urllib3_contextwith a default ofVERIFY_X509_PARTIAL_CHAINandVERIFY_X509_STRICTfor Python 3.13+. (https://github.com/urllib3/urllib3/issues/3571)
This seems to have changed the default behavior of TLS connections made with urllib3, specifically on Python 3.13+ Workarounds include downgrading to Python 3.12, or downgrading urllib3 to 2.3.0.
I suspect the fix for this in the kubernetes libary is to update calls to create_urllib3_context with verify_flags set to the previous default value.
We're also running into this issue with an EKS cluster that was created back in 2019. We need to keep the updates flowing, and it seems like the easiest solution for now is to pin urllib3 to v2.3.0, but that's not a viable long-term solution. We would also appreciate a way to configure urllib3 to use the old behaviour, without disabling TLS verification entirely.
What do you see an acceptable fix as being?
I guess you either can specify urllib3 should be a version <2.4.0, or perhaps expose an easy way to disable this new sticker check in urllib3?
we can specify a version < 2.4.0 for urllib3, please add a brief comment about why that is added.
urllib3 only did this because Python 3.13 does it too. See What’s New In Python 3.13?:
ssl.create_default_context() sets ssl.VERIFY_X509_PARTIAL_CHAIN and ssl.VERIFY_X509_STRICT as default flags.
The actual fix is to ensure the certificate generated by Kubernetes is compliant and is not rejected when those two flags are set. Which is the case for Kubernetes 1.17 and later, apparently.
Recent versions of Python (3.13+) enforce stricter SSL certificate validation and require modern X.509 extensions for the certificate chain.
EKS clusters that were originally created on Kubernetes v1.16 or earlier have a root CA certificate without the Subject Key Identifier (SKID) and Authority Key Identifier (AKI) extensions. Most tools like curl, openssl, or kubectl work fine with such certificates, but Python 3.13+ (and recent urllib3/requests) will fail with a CERTIFICATE_VERIFY_FAILED/Missing Authority Key Identifier error, because they now require these fields to be present for secure chain validation.
If your cluster was created on Kubernetes v1.17 or newer, its CA certificate includes the necessary SKID and AKI, so Python 3.13+ clients are able to connect without any problems.
To summarize:
If your EKS cluster was created on Kubernetes v1.16 or lower, SSL connections from recent Python versions will fail due to missing X.509 extensions in the CA. For clusters created on v1.17 or higher, everything works as expected.
# Old EKS cluster (Kubernetes v1.16 or lower)
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment, Certificate Sign
X509v3 Basic Constraints: critical
CA:TRUE
// (Subject Key Identifier, Authority Key Identifier not present)
# New EKS cluster (Kubernetes v1.17 or higher)
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment, Certificate Sign
X509v3 Basic Constraints: critical
CA:TRUE
X509v3 Subject Key Identifier:
XX:XX:XX:...
// (Authority Key Identifier often not present for root CA, but SKID is present)
I have made a PR to limit the highest urllib3 version allowed with this library: https://github.com/kubernetes-client/python/pull/2417.
An additional problem is that there are two known vulnerabilities that affect any urllib3 versions lower than 2.5.0, with apparently no released backports for 2.3.x:
- https://www.mend.io/vulnerability-database/CVE-2025-50181
- https://www.mend.io/vulnerability-database/CVE-2025-50182
Had the same issue. For me, the "fix" was to use pyenv from https://github.com/pyenv/pyenv to download python 3.12 and try again.
Everything worked after that.
EDIT: yes I was running this against an EKS kubernetes cluster provisioned in ~2019.
If you don't want to downgrade Python or urllib, or disable TLS validation, you can use this workaround when creating the Kubernetes API client:
api_client = ApiClient()
# Ugly workaround to make the OpenSSL TLS validation laxer
# to be compatible to old root CA certificates (created using k8s <= 1.16).
# These are not compliant to openSSL's VERIFY_X509_STRICT that is used per default with Python 3.13.
ctx = ssl.create_default_context()
ctx.verify_flags = ctx.verify_flags & ~ssl.VERIFY_X509_STRICT
api_client.rest_client.pool_manager = urllib3.PoolManager(
num_pools=4, # default value from the kubernetes library that cannot be extracted programmatically.
ssl_context=ctx,
**api_client.rest_client.pool_manager.connection_pool_kw,
)
