kubernetes-client
NO_PROXY/no_proxy env vars not working as expected, as of 34.1.0
What happened (please include outputs or screenshots): Upgrading libraries in one of my images. Last image was built 4 days ago, works fine. We were running kubernetes 33.1.0. Today's image is pulling 34.1.0. KO
Picking apart each package version that changed as of my previous image, I ended up suspecting kubernetes client itself. https://github.com/kubernetes-client/python/compare/v33.1.0...v34.1.0
Obviously, this: https://github.com/kubernetes-client/python/commit/e3b373fc1f31821ab4cd9c288c0263c3bb1b1354#diff-977bedbaf9339b25749bc96a24266c1fa3e99ab521a835562fe84b2ab87b7001
And then again, the rest client, which is the one throwing in my case, looks OK? https://github.com/kubernetes-client/python/blob/master/kubernetes/client/rest.py
That "should_bypass_proxy" ... looks correct ... https://requests.readthedocs.io/en/latest/_modules/requests/utils/
We are using proxies. Ansible would connect public services (eg: aws). Our proxies would only allow for specific names out. Ansible would connect to our clusters directly/no proxy.
Our environment would include:
HTTP_PROXY: http://my-http-proxy:8080
HTTPS_PROXY: http://my-http-proxy:8080
NO_PROXY: github.corp.com,.github.corp.com,localhost,.local,.svc,10.0.0.0/8,127.0.0.0/8,127.0.0.1,::1,.corp.com
http_proxy: http://my-http-proxy:8080
https_proxy: http://my-http-proxy:8080
no_proxy: github.corp.com,.github.corp.com,localhost,.local,.svc,10.0.0.0/8,127.0.0.0/8,127.0.0.1,::1,.corp.com
My cluster FQDNs would follow format <cluster-name>.<where-it-runs>.corp.com
So far, and testing again with my previous images running 33.1.0, connections to AWS exits through proxies, connections to clusters exits directly.
Testing 34.1.0, connection to clusters are mistakenly sent to my proxy, which denies the request.
Forcing HTTPS_PROXY/https_proxy to empty string, when calling my terraform provider: access to cluster OK, but then I broke access to AWS.
I lost that link, but I found somewhere suggesting that some client may handle subdomains differently, so I tried to add corp.com alongside my previous .corp.com, in no_proxy env var. Still no luck.
Then I tried adding the exact cluster FQDN to no_proxy env var: I can still see ansible connecting through my proxies.
What you expected to happen:
domains/subdomains listed in my NO_PROXY/no_proxy should be connected to directly.
How to reproduce it (as minimally and precisely as possible):
Set proxy variables as shown above. Include Kubernetes cluster domain in no_proxy. Switch from 33.1.0 to 34.1.0. check access logs in proxy.
Anything else we need to know?:
Environment:
- Kubernetes version (
kubectl version): 1.29.7 - OS (e.g., MacOS 10.13.6): RHEL9
- Python version (
python --version): 3.9 - Python client version (
pip list | grep kubernetes): 34.1.0
I could work around this re-building my image and pinning kubernetes to 33.1.0
I also found that issue in some controller of mine. That one connects to kubernetes.default.svc, as well as aws/azure APIs, publishing our serviceaccount public signing keys. Running Python 3.12.
Same as my initial issue (a Tekton task running terraform), that controller has environment variables, configuring proxy/no-proxy, such as kubernetes API accesses would remain private, and everything else goes through proxies. Having no_proxy=localhost,127.0.0.1,.local,.internal,.svc,.default,.corp.com,,169.254.169.254,10.96.0.1
Been working fine, up until I merged that last Renovate pull request, upgrading kubernetes. Now I'm getting 403s as controller connects kube API through proxies that don't allow for it.
Traceback (most recent call last):
File "/opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py", line 773, in urlopen
self._prepare_proxy(conn)
File "/opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py", line 1042, in _prepare_proxy
conn.connect()
File "/opt/app-root/lib64/python3.12/site-packages/urllib3/connection.py", line 721, in connect
self._tunnel()
File "/usr/lib64/python3.12/http/client.py", line 981, in _tunnel
raise OSError(f"Tunnel connection failed: {code} {message.strip()}")
OSError: Tunnel connection failed: 403 Forbidden
The above exception was the direct cause of the following exception:
urllib3.exceptions.ProxyError: ('Unable to connect to proxy', OSError('Tunnel connection failed: 403 Forbidden'))
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/app-root/src/app.py", line 25, in <module>
dyn_client = dynamic.DynamicClient(k8s_client)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/app-root/lib64/python3.12/site-packages/kubernetes/dynamic/client.py", line 84, in __init__
self.__discoverer = discoverer(self, cache_file)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/app-root/lib64/python3.12/site-packages/kubernetes/dynamic/discovery.py", line 230, in __init__
Discoverer.__init__(self, client, cache_file)
File "/opt/app-root/lib64/python3.12/site-packages/kubernetes/dynamic/discovery.py", line 55, in __init__
self.__init_cache()
File "/opt/app-root/lib64/python3.12/site-packages/kubernetes/dynamic/discovery.py", line 71, in __init_cache
self._load_server_info()
File "/opt/app-root/lib64/python3.12/site-packages/kubernetes/dynamic/discovery.py", line 146, in _load_server_info
'kubernetes': self.client.request('get', '/version', serializer=just_json)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/app-root/lib64/python3.12/site-packages/kubernetes/dynamic/client.py", line 55, in inner
resp = func(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/app-root/lib64/python3.12/site-packages/kubernetes/dynamic/client.py", line 277, in request
api_response = self.client.call_api(
^^^^^^^^^^^^^^^^^^^^^
File "/opt/app-root/lib64/python3.12/site-packages/kubernetes/client/api_client.py", line 348, in call_api
return self.__call_api(resource_path, method,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/app-root/lib64/python3.12/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
response_data = self.request(
^^^^^^^^^^^^^
File "/opt/app-root/lib64/python3.12/site-packages/kubernetes/client/api_client.py", line 373, in request
return self.rest_client.GET(url,
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/app-root/lib64/python3.12/site-packages/kubernetes/client/rest.py", line 244, in GET
return self.request("GET", url,
^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/app-root/lib64/python3.12/site-packages/kubernetes/client/rest.py", line 217, in request
r = self.pool_manager.request(method, url,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/app-root/lib64/python3.12/site-packages/urllib3/_request_methods.py", line 135, in request
return self.request_encode_url(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/app-root/lib64/python3.12/site-packages/urllib3/_request_methods.py", line 182, in request_encode_url
return self.urlopen(method, url, **extra_kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/app-root/lib64/python3.12/site-packages/urllib3/poolmanager.py", line 633, in urlopen
return super().urlopen(method, url, redirect=redirect, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/app-root/lib64/python3.12/site-packages/urllib3/poolmanager.py", line 443, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py", line 871, in urlopen
return self.urlopen(
^^^^^^^^^^^^^
File "/opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py", line 871, in urlopen
return self.urlopen(
^^^^^^^^^^^^^
File "/opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py", line 871, in urlopen
return self.urlopen(
^^^^^^^^^^^^^
File "/opt/app-root/lib64/python3.12/site-packages/urllib3/connectionpool.py", line 841, in urlopen
retries = retries.increment(
^^^^^^^^^^^^^^^^^^
File "/opt/app-root/lib64/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment
raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='10.96.0.1', port=443): Max retries exceeded with url: /version (Caused by ProxyError('Unable to connect to proxy', OSError('Tunnel connection failed: 403 Forbidden')))
Got same issue, running automated proxy kubernetes build every week, there was no code change on my side and todays build crashed because proxy denied connection to cluster API which but should not go to proxy but direct to API loadbalancer. After downgrading module to 33.1.0 build passed.
There's already a pull request which should fix this bug: https://github.com/kubernetes-client/python/pull/2459
Please check https://github.com/kubernetes-client/python/pull/2459#issuecomment-3486909764 for more info..
Currently if HTTP_PROXY/HTTPS_PROXY and NO_PROXY are set, NO_PROXY proxy wouldn't behave as expect..
Downgrading to v33.1.0 wouldn't honour HTTP_PROXY or HTTPS_PROXY at all, so it works as if NO_PROXY working correctly..
The workaround I have is to manually remove the unnecessary line from https://github.com/kubernetes-client/python/pull/2459
eg. sed -i -e '173d' /usr/local/lib/<python-version>/site-packages/kubernetes/client/configuration.py
tried to leave a comment in the MR with a potential fix: something like the following is needed
BLOCK=""
NOPROXY_BLOCK=""
if [ -z "$NO_PROXY_LINE" ]; then
# self.no_proxy = None is not present → insert full block after self.proxy = None
BLOCK+="${INDENT}# Load proxy from environment variables (if set)\n"
BLOCK+="${INDENT}if os.getenv(\"HTTPS_PROXY\"): self.proxy = os.getenv(\"HTTPS_PROXY\")\n"
BLOCK+="${INDENT}if os.getenv(\"https_proxy\"): self.proxy = os.getenv(\"https_proxy\")\n"
BLOCK+="${INDENT}if os.getenv(\"HTTP_PROXY\"): self.proxy = os.getenv(\"HTTP_PROXY\")\n"
BLOCK+="${INDENT}if os.getenv(\"http_proxy\"): self.proxy = os.getenv(\"http_proxy\")\n"
sed -i "${PROXY_LINE}a $BLOCK" "$CONFIG_FILE"
NOPROXY_BLOCK+="${INDENT}# Load no_proxy from environment variables (if set)\n"
NOPROXY_BLOCK+="${INDENT}if os.getenv(\"NO_PROXY\"): self.no_proxy = os.getenv(\"NO_PROXY\")\n"
NOPROXY_BLOCK+="${INDENT}if os.getenv(\"no_proxy\"): self.no_proxy = os.getenv(\"no_proxy\")"
sed -i "${NO_PROXY_LINE}a $NOPROXY_BLOCK" "$CONFIG_FILE"
echo "Inserted proxy block after 'self.proxy = None'. and no_proxy block after 'self.no_proxy = None'."
else
Downgrading to v33.1.0 wouldn't honour HTTP_PROXY or HTTPS_PROXY at all
Maybe we should point out this behavior is consistent with kubectl/oc clients would work. Doesn't mean you can't use proxies though.
If you want to use a proxy, connecting a kubernetes API, you would just say so, in your kube configuration:
clusters:
- cluster:
proxy-url: http://my-egressproxy:3128
server: https://api.stuff.com:6443
name: foo
This has been working great for years, can be done in python code, and is usually easier to deal with than NO_PROXY schenanigans, when some API needs to exit directly and others would need some specific proxies
@faust64 If you are saying that kubectl doesn't honor $HTTPS_PROXY, you are mistaken. It handles the whole set of variables (http_proxy, HTTP_PROXY, https_proxy, HTTPS_PROXY, no_proxy, NO_PROXY) by virtue of using Go's net/http.
