KubeLibrary icon indicating copy to clipboard operation
KubeLibrary copied to clipboard
verified publisher icon devopsspiral

certificate verify failed issue when using Get Namespaced Pod Exec

Open yannpichon-su opened this issue 3 years ago • 6 comments

When using the Get Namespaced Pod Exec keywork on a k8s cluster using a custom CA, the following error occurs :

ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1129)

Other keywords (Read Namespaced Pod Status, List Namespaced Pod By Pattern ...) are working as expected.

As a quick fix, I'm adding the following line in the _add_api method of the library :

def _add_api(self, reference, class_name):
    self.__dict__[reference] = class_name(self.api_client)
    if not self.cert_validation:
        self.__dict__[reference].api_client.rest_client.pool_manager.connection_pool_kw['cert_reqs'] = ssl.CERT_NONE
        self.__dict__[reference].api_client.configuration.verify_ssl = False

Am I missing something regarding the library configuration ?

Versions :

KubeLibrary: 0.8.0
Python: 3.9.13
Kubernetes: 1.24

KubeLibrary :

Library    KubeLibrary    kube_config=${KUBECONFIG_FILE}    cert_validation=False

KubeLibrary.Get Namespaced Pod Exec    
...    name=my-pod
...    namespace=${namespace}
...    argv_cmd=${command}

yannpichon-su avatar Dec 12 '22 14:12 yannpichon-su

Yeah your workaround is way to go, it does the current setting underneath so it is preferred, Please take a look at v0.8.1 at let me know if it works for you.

m-wcislo avatar Dec 16 '22 15:12 m-wcislo

Thanks for the release @m-wcislo! However, removing the line https://github.com/devopsspiral/KubeLibrary/pull/124/files#diff-65edc7c7cbe7902bb5d18f0b4555bbf308e8cac89745ce248bec177fd9a9b334L279 introduced a regression.

For example, using the keyword List Namespaced Daemon Set leads to the following error:

WARN	Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1129)'))': /k8s/clusters/***redacted***/apis/apps/v1/namespaces/demo/daemonsets?labelSelector=&watch=False
09:30:49.767
WARN	Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1129)'))': /k8s/clusters/***redacted***/apis/apps/v1/namespaces/demo/daemonsets?labelSelector=&watch=False
09:30:49.850
WARN	Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1129)'))': /k8s/clusters/***redacted***/apis/apps/v1/namespaces/demo/daemonsets?labelSelector=&watch=False
09:30:49.929
FAIL	MaxRetryError: HTTPSConnectionPool(host='***redacted***', port=443): Max retries exceeded with url: /k8s/clusters/***redacted***/apis/apps/v1/namespaces/demo/daemonsets?labelSelector=&watch=False (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1129)')))
09:30:49.930
DEBUG	Traceback (most recent call last):
  File "***redacted***/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "***redacted***/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "***redacted***/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
    conn.connect()
  File "***redacted***/.venv/lib/python3.9/site-packages/urllib3/connection.py", line 414, in connect
    self.sock = ssl_wrap_socket(
  File "***redacted***/.venv/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "***redacted***/.venv/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/home/linuxbrew/.linuxbrew/opt/[email protected]/lib/python3.9/ssl.py", line 501, in wrap_socket
    return self.sslsocket_class._create(
  File "/home/linuxbrew/.linuxbrew/opt/[email protected]/lib/python3.9/ssl.py", line 1041, in _create
    self.do_handshake()
  File "/home/linuxbrew/.linuxbrew/opt/[email protected]/lib/python3.9/ssl.py", line 1310, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1129)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "***redacted***/.venv/lib/python3.9/site-packages/KubeLibrary/KubeLibrary.py", line 1378, in list_namespaced_daemon_set
    ret = self.appsv1.list_namespaced_daemon_set(namespace, watch=False, label_selector=label_selector)
  File "***redacted***/.venv/lib/python3.9/site-packages/kubernetes/client/api/apps_v1_api.py", line 3181, in list_namespaced_daemon_set
    return self.list_namespaced_daemon_set_with_http_info(namespace, **kwargs)  # noqa: E501
  File "***redacted***/.venv/lib/python3.9/site-packages/kubernetes/client/api/apps_v1_api.py", line 3296, in list_namespaced_daemon_set_with_http_info
    return self.api_client.call_api(
  File "***redacted***/.venv/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 348, in call_api
    return self.__call_api(resource_path, method,
  File "***redacted***/.venv/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
    response_data = self.request(
  File "***redacted***/.venv/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 373, in request
    return self.rest_client.GET(url,
  File "***redacted***/.venv/lib/python3.9/site-packages/kubernetes/client/rest.py", line 241, in GET
    return self.request("GET", url,
  File "***redacted***/.venv/lib/python3.9/site-packages/kubernetes/client/rest.py", line 214, in request
    r = self.pool_manager.request(method, url,
  File "***redacted***/.venv/lib/python3.9/site-packages/urllib3/request.py", line 74, in request
    return self.request_encode_url(
  File "***redacted***/.venv/lib/python3.9/site-packages/urllib3/request.py", line 96, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
  File "***redacted***/.venv/lib/python3.9/site-packages/urllib3/poolmanager.py", line 376, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "***redacted***/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 815, in urlopen
    return self.urlopen(
  File "***redacted***/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 815, in urlopen
    return self.urlopen(
  File "***redacted***/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 815, in urlopen
    return self.urlopen(
  File "***redacted***/.venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
  File "***redacted***/.venv/lib/python3.9/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='***redacted***', port=443): Max retries exceeded with url: /k8s/clusters/***redacted***/apis/apps/v1/namespaces/demo/daemonsets?labelSelector=&watch=False (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1129)')))

Could it be reverted please?

fdaguin avatar Dec 19 '22 08:12 fdaguin

ok, reverted in 0.8.2, need to take closer look.

m-wcislo avatar Dec 19 '22 09:12 m-wcislo

Thank you! From our tests, keeping both

    if not self.cert_validation:
        self.__dict__[reference].api_client.rest_client.pool_manager.connection_pool_kw['cert_reqs'] = ssl.CERT_NONE
        self.__dict__[reference].api_client.configuration.verify_ssl = False

was working as expected.

fdaguin avatar Dec 19 '22 09:12 fdaguin

Actually it is reverted in 0.8.3. yeah need to track down how the setting is propagated, latest setting should set both

m-wcislo avatar Dec 19 '22 10:12 m-wcislo

Ok, did some testing and cannot really reproduce it on my end, so have couple questions:

  1. Did it worked in the past for you and is just broken recently? If yes what changes were made?
  2. Your kubeconfig doesn't have :
clusters:
- cluster:
    certificate-authority-data: ?

This is the easiest way to make your connection trust the API server cert, without need of use cert_validation=False 3. Are the failing (Get Namespaced Pod Exec) tests are executed somehow differently (different kubeconfig, settings etc.)than the ones that are passing (List Namespaced Daemon Set)? 4. Would it be possible to list packages versions you are using (pip freeze) at least for things in reuirements.txt

m-wcislo avatar Dec 31 '22 11:12 m-wcislo