Kubernetes client exception with connection refused error

[hardys666]

Our tests had been running fine till we started running into the following issue where the kubernetes client websocket call will terminate with an exception:

channel = stream(api.connect_get_namespaced_pod_exec,

/usr/lib/python3.8/site-packages/kubernetes/stream/stream.py:35: in _websocket_request return api_method(*args, **kwargs) /usr/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py:994: in connect_get_namespaced_pod_exec return self.connect_get_namespaced_pod_exec_with_http_info(name, namespace, **kwargs) # noqa: E501 /usr/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py:1101: in connect_get_namespaced_pod_exec_with_http_info return self.api_client.call_api( /usr/lib/python3.8/site-packages/kubernetes/client/api_client.py:348: in call_api return self.__call_api(resource_path, method, /usr/lib/python3.8/site-packages/kubernetes/client/api_client.py:180: in __call_api response_data = self.request(

---

configuration = <kubernetes.client.configuration.Configuration object at 0x7f192844cb20> _method = 'GET' url = 'wss://api.ndal-ocp2.ve.pd.f5net.com:6443/api/v1/namespaces/spk-test-automation/pods/ans-client-ci6492742-127/exec?com...command=2001%3A0db8%3A0%3A127%3A%3A100&command=-t&command=10&command=-P8&stderr=True&stdin=False&stdout=True&tty=False' kwargs = {'_preload_content': False, '_request_timeout': None, 'body': None, 'headers': {'Accept': '/', 'User-Agent': 'OpenAPI-Generator/25.3.0/python', 'authorization': 'Bearer sha256~bgMewSf5E2EdW_h4EYLI4Ju-I21kPKVzK5tRTvqFgtg'}, ...} headers = {'Accept': '/', 'User-Agent': 'OpenAPI-Generator/25.3.0/python', 'authorization': 'Bearer sha256~bgMewSf5E2EdW_h4EYLI4Ju-I21kPKVzK5tRTvqFgtg'} _request_timeout = None, _preload_content = False, capture_all = True def websocket_call(configuration, _method, url, **kwargs): """An internal function to be called in api-client when a websocket connection is required. method, url, and kwargs are the parameters of apiClient.request method."""

    url = get_websocket_url(url, kwargs.get("query_params"))
    headers = kwargs.get("headers")
    _request_timeout = kwargs.get("_request_timeout", 60)
    _preload_content = kwargs.get("_preload_content", True)
    capture_all = kwargs.get("capture_all", True)

    try:
        client = WSClient(configuration, url, headers, capture_all)
        if not _preload_content:
            return client
        client.run_forever(timeout=_request_timeout)
        return WSResponse('%s' % ''.join(client.read_all()))
    except (Exception, KeyboardInterrupt, SystemExit) as e:

      raise ApiException(status=0, reason=str(e))

E kubernetes.client.exceptions.ApiException: (0) E Reason: [Errno 111] Connection refused /usr/lib/python3.8/site-packages/kubernetes/stream/ws_client.py:525: ApiException

These exceptions happen at random (not for any specific or every test) and across different kubernetes environment/clusters. They started happening somewhere around early Feb, 2023. (not sure if its related to any fixes/changes).

Below is what we use in our code: `def k8s_exec(command, pod, namespace, api): channel = stream(api.connect_get_namespaced_pod_exec, pod, namespace, command=command, stderr=True, stdin=False, stdout=True, tty=False, _preload_content=False) channel.run_forever(timeout=60) err = yaml.safe_load(channel.read_channel(ERROR_CHANNEL)) if err is None: raise AssertionError(f'The command "{command}" was timeouted') rc = 0 if err['status'] == 'Success' else 1 stdout = channel.read_stdout() stderr = channel.read_stderr()

return rc, stdout, stderr`

Not sure if this is similar to: 1841

What you expected to happen: Connection is retained.

Environment:

• Kubernetes version (kubectl version): Client Version: version.Info{Major:"1", Minor:"20" Server Version: version.Info{Major:"1", Minor:"23"
• OS (e.g., MacOS 10.13.6): Linux
• Python version (python --version): 2.7.5