etcd container kill scenario doesn't wait for the apiserver to become available

[Noreen21]

2023-04-21 16:44:27,328 [INFO] Executing scenarios for iteration 0
2023-04-21 16:44:27,328 [INFO] connection set up
127.0.0.1 - - [21/Apr/2023 16:44:27] "GET / HTTP/1.1" 200 -
2023-04-21 16:44:27,329 [INFO] response RUN
2023-04-21 16:44:27,330 [INFO] Running container scenarios
2023-04-21 16:44:27,843 [INFO] Killing container etcd in pod etcd-e28-h35-r750 (ns openshift-etcd)
2023-04-21 16:44:28,140 [INFO] Scenario kill etcd container successfully injected
^[[B2023-04-21 16:45:13,931 [WARNING] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', RemoteDisconnected('Remote end closed connection withou$ response'))': /api/v1/namespaces/openshift-etcd/pods?pretty=True
2023-04-21 16:45:14,033 [WARNING] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))': /a$ i/v1/namespaces/openshift-etcd/pods?pretty=True
2023-04-21 16:45:14,034 [WARNING] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f991506afd0>: Failed to est$ blish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/openshift-etcd/pods?pretty=True
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 174, in _new_conn
conn = connection.create_connection(
File "/usr/local/lib/python3.9/site-packages/urllib3/util/connection.py", line 95, in create_connection
raise err
File "/usr/local/lib/python3.9/site-packages/urllib3/util/connection.py", line 85, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 386, in _make_request
self._validate_conn(conn)
File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
conn.connect()
File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 363, in connect
self.sock = conn = self._new_conn() File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 186, in _new_conn raise NewConnectionError( urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f9915029520>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/root/sno/e28-h35-r750/krkn/run_kraken.py", line 403, in main(options.cfg) File "/root/sno/e28-h35-r750/krkn/run_kraken.py", line 214, in main failed_post_scenarios = pod_scenarios.container_run( File "/root/sno/e28-h35-r750/krkn/kraken/pod_scenarios/setup.py", line 92, in container_run failed_post_scenarios = check_failed_containers( File "/root/sno/e28-h35-r750/krkn/kraken/pod_scenarios/setup.py", line 191, in check_failed_containers pod_output = kubecli.get_pod_info(killed_container[0], killed_container[1]) File "/root/sno/e28-h35-r750/krkn/kraken/kubernetes/client.py", line 544, in get_pod_info pod_exists = check_if_pod_exists(name=name, namespace=namespace) File "/root/sno/e28-h35-r750/krkn/kraken/kubernetes/client.py", line 721, in check_if_pod_exists pod_list = list_pods(namespace=namespace) File "/root/sno/e28-h35-r750/krkn/kraken/kubernetes/client.py", line 209, in list_pods ret = cli.list_namespaced_pod(namespace, pretty=True) File "/usr/local/lib/python3.9/site-packages/kubernetes/client/api/core_v1_api.py", line 15697, in list_namespaced_pod return self.list_namespaced_pod_with_http_info(namespace, **kwargs) # noqa: E501 File "/usr/local/lib/python3.9/site-packages/kubernetes/client/api/core_v1_api.py", line 15812, in list_namespaced_pod_with_http_info return self.api_client.call_api( File "/usr/local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 348, in call_api return self.__call_api(resource_path, method, File "/usr/local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 180, in __call_api response_data = self.request( File "/usr/local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 373, in request return self.rest_client.GET(url, File "/usr/local/lib/python3.9/site-packages/kubernetes/client/rest.py", line 241, in GET return self.request("GET", url, File "/usr/local/lib/python3.9/site-packages/kubernetes/client/rest.py", line 214, in request r = self.pool_manager.request(method, url, File "/usr/local/lib/python3.9/site-packages/urllib3/request.py", line 74, in request return self.request_encode_url( File "/usr/local/lib/python3.9/site-packages/urllib3/request.py", line 96, in request_encode_url return self.urlopen(method, url, **extra_kw) File "/usr/local/lib/python3.9/site-packages/urllib3/poolmanager.py", line 376, in urlopen response = conn.urlopen(method, u.request_uri, **kw) File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 815, in urlopen return self.urlopen( File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 815, in urlopen return self.urlopen( File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 815, in urlopen return self.urlopen( File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 787, in urlopen retries = retries.increment( File "/usr/local/lib/python3.9/site-packages/urllib3/util/retry.py", line 592, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.e28-h35-r750.example.com', port=6443): Max retries exceeded with url: /api/v1/namespaces/openshift-etcd/pods?pretty=True (Caused by NewConnectionError('<urllib3.connection.HT TPSConnection object at 0x7f9915029520>: Failed to establish a new connection: [Errno 111] Connection refused'))

[Noreen21]

Also cerberus doesn't publish the apiserver failure status