Using kubernetes-client with Teleport auth proxy does not seem to work

[mattmelgard]

What happened (please include outputs or screenshots):

Our Kubernetes clusters have Teleport agents installed on them as a means to manage secure/auditable access for our engineers. The agents basically act as an auth proxy for all requests to the k8s API. This works just fine with kubectl but does not seem to work out of the box with the Python Kubernetes Client for whatever reason.

For example, when trying to access the cluster using a code snippet like the following:

from kubernetes import client
from kubernetes import config as k8s_config
...

k8s_config.load_kube_config(context='my-teleport-context-name')
api_client = client.CoreV1Api()
services = api_client.list_namespaced_service(namespace='some-namespace')
...

I get back the error (with infrastructure/tenant specific details redacted/replaced)

Traceback (most recent call last):
  File "/Users/mattmelgard/.pyenv/versions/3.9.0/envs/syndio-backend/lib/python3.9/site-packages/syndioctlpy/syndioctl.py", line 64, in handle_errors
    return func(*args, **kwargs)
  File "/Users/mattmelgard/.pyenv/versions/3.9.0/envs/syndio-backend/lib/python3.9/site-packages/syndioctlpy/beta.py", line 286, in port_forward
    k8s.port_forward()
  File "/Users/mattmelgard/.pyenv/versions/3.9.0/envs/syndio-backend/lib/python3.9/site-packages/syndioctlpy/kubernetes.py", line 68, in port_forward
    services = api_client.list_namespaced_service(namespace=namespace)
  File "/Users/mattmelgard/.pyenv/versions/3.9.0/envs/syndio-backend/lib/python3.9/site-packages/kubernetes/client/api/core_v1_api.py", line 16517, in list_namespaced_service
    return self.list_namespaced_service_with_http_info(namespace, **kwargs)  # noqa: E501
  File "/Users/mattmelgard/.pyenv/versions/3.9.0/envs/syndio-backend/lib/python3.9/site-packages/kubernetes/client/api/core_v1_api.py", line 16632, in
list_namespaced_service_with_http_info
    return self.api_client.call_api(
  File "/Users/mattmelgard/.pyenv/versions/3.9.0/envs/syndio-backend/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 348, in call_api
    return self.__call_api(resource_path, method,
  File "/Users/mattmelgard/.pyenv/versions/3.9.0/envs/syndio-backend/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
    response_data = self.request(
  File "/Users/mattmelgard/.pyenv/versions/3.9.0/envs/syndio-backend/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 373, in request
    return self.rest_client.GET(url,
  File "/Users/mattmelgard/.pyenv/versions/3.9.0/envs/syndio-backend/lib/python3.9/site-packages/kubernetes/client/rest.py", line 241, in GET
    return self.request("GET", url,
  File "/Users/mattmelgard/.pyenv/versions/3.9.0/envs/syndio-backend/lib/python3.9/site-packages/kubernetes/client/rest.py", line 214, in request
    r = self.pool_manager.request(method, url,
  File "/Users/mattmelgard/.pyenv/versions/3.9.0/envs/syndio-backend/lib/python3.9/site-packages/urllib3/request.py", line 74, in request
    return self.request_encode_url(
  File "/Users/mattmelgard/.pyenv/versions/3.9.0/envs/syndio-backend/lib/python3.9/site-packages/urllib3/request.py", line 96, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
  File "/Users/mattmelgard/.pyenv/versions/3.9.0/envs/syndio-backend/lib/python3.9/site-packages/urllib3/poolmanager.py", line 375, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/Users/mattmelgard/.pyenv/versions/3.9.0/envs/syndio-backend/lib/python3.9/site-packages/urllib3/connectionpool.py", line 783, in urlopen
    return self.urlopen(
  File "/Users/mattmelgard/.pyenv/versions/3.9.0/envs/syndio-backend/lib/python3.9/site-packages/urllib3/connectionpool.py", line 783, in urlopen
    return self.urlopen(
  File "/Users/mattmelgard/.pyenv/versions/3.9.0/envs/syndio-backend/lib/python3.9/site-packages/urllib3/connectionpool.py", line 783, in urlopen
    return self.urlopen(
  File "/Users/mattmelgard/.pyenv/versions/3.9.0/envs/syndio-backend/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/Users/mattmelgard/.pyenv/versions/3.9.0/envs/syndio-backend/lib/python3.9/site-packages/urllib3/util/retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='something.teleport.sh', port=443): Max retries exceeded with url: /api/v1/namespaces/mattmelgard/services (Caused by
SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1122)')))

Disabling SSL verification in order to validate that client is even able to connect to the cluster itself with the following code:

from kubernetes import client
from kubernetes import config as k8s_config
...

configuration = client.Configuration()
configuration.verify_ssl = False
client.Configuration.set_default(configuration)

k8s_config.load_kube_config(context='my-teleport-context-name')
api_client = client.CoreV1Api()
services = api_client.list_namespaced_service(namespace='some-namespace', client_configuration=configuration)
...

I instead get back this error that seems to indicate that the client isn't even picking up the correct configuration for the hostname or port of the cluster (note the localhost:80 address being used in the error):

Traceback (most recent call last):
  File "/Users/mattmelgard/.pyenv/versions/3.9.0/envs/syndio-backend/lib/python3.9/site-packages/syndioctlpy/syndioctl.py", line 64, in handle_errors
    return func(*args, **kwargs)
  File "/Users/mattmelgard/.pyenv/versions/3.9.0/envs/syndio-backend/lib/python3.9/site-packages/syndioctlpy/beta.py", line 286, in port_forward
    k8s.port_forward()
  File "/Users/mattmelgard/.pyenv/versions/3.9.0/envs/syndio-backend/lib/python3.9/site-packages/syndioctlpy/kubernetes.py", line 68, in port_forward
    services = api_client.list_namespaced_service(namespace=namespace)
  File "/Users/mattmelgard/.pyenv/versions/3.9.0/envs/syndio-backend/lib/python3.9/site-packages/kubernetes/client/api/core_v1_api.py", line 16517, in list_namespaced_service
    return self.list_namespaced_service_with_http_info(namespace, **kwargs)  # noqa: E501
  File "/Users/mattmelgard/.pyenv/versions/3.9.0/envs/syndio-backend/lib/python3.9/site-packages/kubernetes/client/api/core_v1_api.py", line 16632, in
list_namespaced_service_with_http_info
    return self.api_client.call_api(
  File "/Users/mattmelgard/.pyenv/versions/3.9.0/envs/syndio-backend/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 348, in call_api
    return self.__call_api(resource_path, method,
  File "/Users/mattmelgard/.pyenv/versions/3.9.0/envs/syndio-backend/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
    response_data = self.request(
  File "/Users/mattmelgard/.pyenv/versions/3.9.0/envs/syndio-backend/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 373, in request
    return self.rest_client.GET(url,
  File "/Users/mattmelgard/.pyenv/versions/3.9.0/envs/syndio-backend/lib/python3.9/site-packages/kubernetes/client/rest.py", line 241, in GET
    return self.request("GET", url,
  File "/Users/mattmelgard/.pyenv/versions/3.9.0/envs/syndio-backend/lib/python3.9/site-packages/kubernetes/client/rest.py", line 214, in request
    r = self.pool_manager.request(method, url,
  File "/Users/mattmelgard/.pyenv/versions/3.9.0/envs/syndio-backend/lib/python3.9/site-packages/urllib3/request.py", line 74, in request
    return self.request_encode_url(
  File "/Users/mattmelgard/.pyenv/versions/3.9.0/envs/syndio-backend/lib/python3.9/site-packages/urllib3/request.py", line 96, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
  File "/Users/mattmelgard/.pyenv/versions/3.9.0/envs/syndio-backend/lib/python3.9/site-packages/urllib3/poolmanager.py", line 375, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/Users/mattmelgard/.pyenv/versions/3.9.0/envs/syndio-backend/lib/python3.9/site-packages/urllib3/connectionpool.py", line 783, in urlopen
    return self.urlopen(
  File "/Users/mattmelgard/.pyenv/versions/3.9.0/envs/syndio-backend/lib/python3.9/site-packages/urllib3/connectionpool.py", line 783, in urlopen
    return self.urlopen(
  File "/Users/mattmelgard/.pyenv/versions/3.9.0/envs/syndio-backend/lib/python3.9/site-packages/urllib3/connectionpool.py", line 783, in urlopen
    return self.urlopen(
  File "/Users/mattmelgard/.pyenv/versions/3.9.0/envs/syndio-backend/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/Users/mattmelgard/.pyenv/versions/3.9.0/envs/syndio-backend/lib/python3.9/site-packages/urllib3/util/retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=80): Max retries exceeded with url: /api/v1/namespaces/mattmelgard/services (Caused by
NewConnectionError('<urllib3.connection.HTTPConnection object at 0x10a4a2220>: Failed to establish a new connection: [Errno 61] Connection refused'))

What you expected to happen:

The Python Kubernetes Client can connect to the cluster the same way that kubectl would when keeping all configuration the same between the two.

How to reproduce it (as minimally and precisely as possible):

Example provided above with the added requirement of a teleport cluster/k8s agent to point things at, though the implementation of their specific credential plugin does not seem to be the root of the issue since kubectl commands work just fine as already mentioned.

Anything else we need to know?:

The Teleport client appears to configure an exec credential to generate temporary credentials for access to the cluster. The configuration for that looks something like this (with infrastructure/tenant specific details redacted/replaced)

- name: something.teleport.sh-my-cluster
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - kube
      - credentials
      - --kube-cluster=my-cluster
      - --teleport-cluster=something.teleport.sh
      - --proxy=something.teleport.sh:443
      command: /usr/local/bin/tsh
      env: null
      interactiveMode: IfAvailable
      provideClusterInfo: false

Environment:

• Kubernetes version (kubectl version): 1.27.3
• OS (e.g., MacOS 10.13.6): MacOS 13.2.1
• Python version (python --version): 3.9.0
• Python client version (pip list | grep kubernetes): 26.1.0

[mattmelgard]

Just as an FYI, using the branch for this PR seems to solve the problem and it appears to be ready to merge. @roycaihw is that something that you or others might be able to review/merge soon?

[roycaihw]

Thanks for pointing that out. We are looking at the PR and hopefully it will be merged soon

[vishal-ramesh2]

Is there an ETA for this to be released?

[jamshidyerzakov]

Any news regarding this issue or is there any temporary workaround for now?

Same error while using ansible kubernetes.core.k8s module via connection through teleport kubernetes agent which uses python client:

"msg": "Failed to get client due to HTTPSConnectionPool(host='teleport.example.com, port=443): Max retries exceeded with url: /version (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)')))",

Tried to ignore tls using "insecure-skip-tls-verify": true. Didn't help: "msg": "Failed to get client due to 404\nReason: Not Found

[jtsai-quid]

I used to have this issue in 26.1.0, I just upgrade to 28.1.0 and now it is resolved.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale