python icon indicating copy to clipboard operation
python copied to clipboard

Kubernetes Clients breaks with urllib3 2.6.0

Open potiuk opened this issue 3 weeks ago • 13 comments

What happened (please include outputs or screenshots):

When urllib3 2.6.0 is used with kubernetes client, it fails because internally it uses "getheaders()" which have been deprecated in 2.0.0 and removed in 2.6.0 .

The offending line is:

https://github.com/kubernetes-client/python/blob/a49d85dbc593e083974099e766809c7b13b243de/kubernetes/client/exceptions.py#L91

Possibly there are other pleaces.

When there is an API exception returned and urllib 2.6.0 you get an exception:

../.venv/lib/python3.10/site-packages/kubernetes/client/api/core_v1_api.py:23999: in read_namespaced_pod
    return self.read_namespaced_pod_with_http_info(name, namespace, **kwargs)  # noqa: E501
../.venv/lib/python3.10/site-packages/kubernetes/client/api/core_v1_api.py:24086: in read_namespaced_pod_with_http_info
    return self.api_client.call_api(
../.venv/lib/python3.10/site-packages/kubernetes/client/api_client.py:348: in call_api
    return self.__call_api(resource_path, method,
../.venv/lib/python3.10/site-packages/kubernetes/client/api_client.py:180: in __call_api
    response_data = self.request(
../.venv/lib/python3.10/site-packages/kubernetes/client/api_client.py:373: in request
    return self.rest_client.GET(url,
../.venv/lib/python3.10/site-packages/kubernetes/client/rest.py:244: in GET
    return self.request("GET", url,
../.venv/lib/python3.10/site-packages/kubernetes/client/rest.py:238: in request
    raise ApiException(http_resp=r)
../.venv/lib/python3.10/site-packages/kubernetes/client/exceptions.py:91: in __init__
    self.headers = http_resp.getheaders()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <kubernetes.client.rest.RESTResponse object at 0x7f0b51d92c50>

    def getheaders(self):
        """Returns a dictionary of the response headers."""
>       return self.urllib3_response.getheaders()
E       AttributeError: 'HTTPResponse' object has no attribute 'getheaders'. Did you mean: 'headers'?

What you expected to happen:

No crash on missing attribute.

How to reproduce it (as minimally and precisely as possible):

I have no minimal reproducer - it started to fail in our CI: https://github.com/apache/airflow/actions/runs/19969105384/job/57271505761#step:7:963 - however the error is pretty obvious:

  • install kubernetes-client
  • install urllib3 2.6.0
  • make any call that raises APIException -> crash.

Anything else we need to know?:

Environment:

  • Kubernetes version (kubectl version): v1.30.13 for example (crashes for all of them)
  • OS (e.g., MacOS 10.13.6): Debian Bookworm
  • Python version (python --version) 3.10.19
  • Python client version (pip list | grep kubernetes) kubernetes==33.1.0

potiuk avatar Dec 05 '25 19:12 potiuk

#2169, #2280, urllib3/urllib3#3731 There was a patch in #2175 but it seems to only touch kubernetes/client/exceptions.py but not kubernetes/client/rest.py

cg505 avatar Dec 05 '25 22:12 cg505

Also this issue does not allow anyone who would like to use k8s client to upgrafde to 2.6.0 that contains fixes to important vulnerabilities

Specifically those two which are assessed as "high" severity - 8.9/10 both

  • https://nvd.nist.gov/vuln/detail/CVE-2025-66471
  • https://nvd.nist.gov/vuln/detail/CVE-2025-66418

potiuk avatar Dec 07 '25 10:12 potiuk

Note that urllib3 just released a version 2.6.1 that restores the removed methods as a short-term fix.

glennmatthews avatar Dec 08 '25 17:12 glennmatthews

If (again as a shortish-term measure) the urllib3 line in requirements.txt were to include a !=2.6.0 constraint, I think that could save people some pain.

david-prior avatar Dec 08 '25 17:12 david-prior

Note that urllib3 just released a version 2.6.1 that restores the removed methods as a short-term fix.

Yep. Also there is an issue in openapi-generator that I opened, which is possibly how the code was generated, so maybe if that will be fixed, the code can be just regenerated. https://github.com/OpenAPITools/openapi-generator/issues/22514

This issue waits for someone who is more Java-ish than I am so maybe someone looking at the issue here can volunteer and fix it.

potiuk avatar Dec 08 '25 17:12 potiuk

If (again as a shortish-term measure) the urllib3 line in requirements.txt were to include a !=2.6.0 constraint, I think that could save people some pain.

I suggested the urllib3 team to yank 2.6.0 and if they will - it might be not needed.

potiuk avatar Dec 08 '25 17:12 potiuk

Please provide a working version with urllib3 version !=2.6.0 asap. I am not allowed to deploy our tooling container with high CVE scores, which basically is the cornerstone of our automation.

Thanks !

torbenaa avatar Dec 11 '25 11:12 torbenaa

Please provide a working version with urllib3 version !=2.6.0 asap. I am not allowed to deploy our tooling container with high CVE scores, which basically is the cornerstone of our automation.

Thanks !

In case you missed it, 2.6.1 has been released by urllib and it works with this client - maybe you missed it ?

potiuk avatar Dec 11 '25 12:12 potiuk

Please provide a working version with urllib3 version !=2.6.0 asap. I am not allowed to deploy our tooling container with high CVE scores, which basically is the cornerstone of our automation. Thanks !

In case you missed it, 2.6.1 has been released by urllib and it works with this client - maybe you missed it ?

Actually - it looks like 2.6.2 is released also !

But kubernetes still requires other versions :

$ pip install -U urllib3
Requirement already satisfied: urllib3 in ./venv/lib/python3.10/site-packages (2.3.0)
Collecting urllib3
  Downloading urllib3-2.6.2-py3-none-any.whl (131 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 131.2/131.2 kB 2.7 MB/s eta 0:00:00
Installing collected packages: urllib3
  Attempting uninstall: urllib3
    Found existing installation: urllib3 2.3.0
    Uninstalling urllib3-2.3.0:
      Successfully uninstalled urllib3-2.3.0
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
kubernetes 34.1.0 requires urllib3<2.4.0,>=1.24.2, but you have urllib3 2.6.2 which is incompatible.
Successfully installed urllib3-2.6.2

torbenaa avatar Dec 11 '25 22:12 torbenaa

But kubernetes still requires other versions :

kubernetes 33.1.0 has this (and it works fine at least in airflow tests with urrlib 2.6.1+):

Requires-Dist: urllib3>=1.24.2

But of course kubernetes team should likely fix it of course.

potiuk avatar Dec 11 '25 22:12 potiuk

But kubernetes still requires other versions :

kubernetes 33.1.0 has this (and it works fine at least in airflow tests with urrlib 2.6.1+):

Requires-Dist: urllib3>=1.24.2

But of course kubernetes team should likely fix it of course.

Thanks for the reply !

So you are suggesting to use 33.1.0 with urllib3 v 2.6.2 until a new release of kubernetes ?

torbenaa avatar Dec 11 '25 22:12 torbenaa

So you are suggesting to use 33.1.0 with urllib3 v 2.6.2 until a new release of kubernetes ?

Well. I am not suggesting it, just telling you that it works for us in Airflow - this is the version of k8s our automated resolution figures out with uv's highestResolution. Whether it will work for you - only you and k8s client maintainers cna tell - and I am not one of them

potiuk avatar Dec 11 '25 22:12 potiuk

So you are suggesting to use 33.1.0 with urllib3 v 2.6.2 until a new release of kubernetes ?

Well. I am not suggesting it, just telling you that it works for us in Airflow - this is the version of k8s our automated resolution figures out with uv's highestResolution. Whether it will work for you - only you and k8s client maintainers cna tell - and I am not one of them

Sorry - the wording came out wrong. I understand that this is an issue that needs to be fixed by the maintainers. Thanks you for your input. I will try to get a workaround deployed today. I guess we are in the same boat :)

torbenaa avatar Dec 12 '25 09:12 torbenaa