router-perf reschedule monitoring not fully completed after test exit

[qiliRedHat]

Run router-perf-v2 test, the test reschedule monitoring stack to infra nodes before test finish. https://github.com/cloud-bulldozer/e2e-benchmarking/blob/1c620ca2f0862c7f7b0bddb032e311bfb0a3553b/workloads/router-perf-v2/common.sh#L135-L143

Mar 16 08:04:12 UTC 2023 Re-scheduling monitoring stack to infra nodes[0m
configmap/cluster-monitoring-config configured
deployment "cluster-monitoring-operator" successfully rolled out
statefulset rolling update complete 2 pods at revision prometheus-k8s-689d496649...

After the router-perf-v2 test script ingress-performance.sh exit, run Cerberus there are still some failed containers.

2023-03-16 08:06:34,743 [INFO] Iteration 1: Failed pods and components
2023-03-16 08:06:34,743 [INFO] openshift-monitoring: ['alertmanager-main-0', 'prometheus-adapter-65f8bdf5d5-t9tnv', 'prometheus-k8s-1', 'prometheus-adapter-65f8bdf5d5-6pb5r']
2023-03-16 08:06:34,743 [INFO] Failed containers in alertmanager-main-0: ['alertmanager', 'alertmanager-proxy', 'config-reloader', 'kube-rbac-proxy', 'kube-rbac-proxy-metric', 'prom-label-proxy']
2023-03-16 08:06:34,743 [INFO] Failed containers in prometheus-adapter-65f8bdf5d5-6pb5r: ['prometheus-adapter']
2023-03-16 08:06:34,743 [INFO] Failed containers in prometheus-adapter-65f8bdf5d5-t9tnv: ['prometheus-adapter']
2023-03-16 08:06:34,743 [INFO] Failed containers in prometheus-k8s-1: ['config-reloader', 'kube-rbac-proxy', 'kube-rbac-proxy-thanos', 'prometheus', 'prometheus-proxy', 'thanos-sidecar', 'init-config-reloader']

[container "alertmanager" in pod "alertmanager-main-0" is waiting to start: ContainerCreating, previous terminated container "alertmanager" in pod "alertmanager-main-0" not found, previous terminated container "config-reloader" in pod "alertmanager-main-0" not found, container "config-reloader" in pod "alertmanager-main-0" is waiting to start: ContainerCreating, container "alertmanager-proxy" in pod "alertmanager-main-0" is waiting to start: ContainerCreating, previous terminated container "alertmanager-proxy" in pod "alertmanager-main-0" not found, container "kube-rbac-proxy" in pod "alertmanager-main-0" is waiting to start: ContainerCreating, previous terminated container "kube-rbac-proxy" in pod "alertmanager-main-0" not found, container "kube-rbac-proxy-metric" in pod "alertmanager-main-0" is waiting to start: ContainerCreating, previous terminated container "kube-rbac-proxy-metric" in pod "alertmanager-main-0" not found, previous terminated container "prom-label-proxy" in pod "alertmanager-main-0" not found, container "prom-label-proxy" in pod "alertmanager-main-0" is waiting to start: ContainerCreating], one or more errors occurred while gathering container data for pod prometheus-adapter-78d8b6cd95-7rdwf:
pods "prometheus-adapter-78d8b6cd95-7rdwf" not found]

Those containers needs some time to be up and running, I hope to add some check to ensure they are up and running before ingress-performance.sh exit.