cerberus
cerberus copied to clipboard
Published
20 hours ago •
krkn-chaos
krkn-chaos
Not catching when nodes are Ready,SchedulingDisabled
Cerberus run passes but 2 nodes are SchedulingDisabled, should catch this as a failure or make it an option
Node has taint: Taints: node.kubernetes.io/unschedulable:NoSchedule
07-12 15:50:03.255
07-12 15:50:03.255 _
07-12 15:50:03.255 ___ ___ _ __| |__ ___ _ __ _ _ ___
07-12 15:50:03.255 / __/ _ \ '__| '_ \ / _ \ '__| | | / __|
07-12 15:50:03.255 | (_| __/ | | |_) | __/ | | |_| \__ \
07-12 15:50:03.255 \___\___|_| |_.__/ \___|_| \__,_|___/
07-12 15:50:03.255
07-12 15:50:03.255
07-12 15:50:03.839 Error: unknown flag: --duration
07-12 15:50:03.839 See 'oc create --help' for usage.
07-12 15:50:04.397 2023-07-12 19:50:04,235 [WARNING] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:2633)'))': /api/v1/namespaces/openshift-kube-apiserver-operator/pods?pretty=True&limit=100
07-12 15:50:04.397 2023-07-12 19:50:04,237 [WARNING] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))': /api/v1/namespaces/openshift-user-workload-monitoring/pods?pretty=True&limit=100
07-12 15:50:04.397 2023-07-12 19:50:04,237 [WARNING] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record mac (_ssl.c:2633)'))': /api/v1/namespaces/openshift-machine-api/pods?pretty=True&limit=100
07-12 15:50:11.055 2023-07-12 19:50:09,944 [INFO] Iteration 1: No Terminating Namespaces status: True
07-12 15:50:11.055 2023-07-12 19:50:09,966 [INFO] Iteration 1: Node status: True
07-12 15:50:11.055 2023-07-12 19:50:10,174 [INFO] Iteration 1: Cluster Operator status: True
07-12 15:50:11.055 2023-07-12 19:50:10,206 [INFO] Iteration 1: openshift-user-workload-monitoring: True
07-12 15:50:11.055 2023-07-12 19:50:10,207 [INFO] Iteration 1: openshift-ovirt-infra: True
07-12 15:50:11.055 2023-07-12 19:50:10,228 [INFO] Iteration 1: openshift-host-network: True
07-12 15:50:11.055 2023-07-12 19:50:10,230 [INFO] Iteration 1: openshift-kube-apiserver-operator: True
07-12 15:50:11.055 2023-07-12 19:50:10,253 [INFO] Iteration 1: openshift-insights: True
07-12 15:50:11.055 2023-07-12 19:50:10,263 [INFO] Iteration 1: openshift-machine-api: True
07-12 15:50:11.055 2023-07-12 19:50:10,271 [INFO] Iteration 1: openshift-cluster-machine-approver: True
07-12 15:50:11.055 2023-07-12 19:50:10,280 [INFO] Iteration 1: openshift-cluster-version: True
07-12 15:50:11.055 2023-07-12 19:50:10,293 [INFO] Iteration 1: openshift-service-ca-operator: True
07-12 15:50:11.055 2023-07-12 19:50:10,296 [INFO] Iteration 1: openshift-kni-infra: True
07-12 15:50:11.055 2023-07-12 19:50:10,311 [INFO] Iteration 1: openshift-apiserver-operator: True
07-12 15:50:11.055 2023-07-12 19:50:10,318 [INFO] Iteration 1: openshift-infra: True
07-12 15:50:11.055 2023-07-12 19:50:10,343 [INFO] Iteration 1: openshift-kube-storage-version-migrator: True
07-12 15:50:11.055 2023-07-12 19:50:10,347 [INFO] Iteration 1: openshift-cloud-credential-operator: True
07-12 15:50:11.055 2023-07-12 19:50:10,354 [INFO] Iteration 1: openshift-ingress: True
07-12 15:50:11.055 2023-07-12 19:50:10,383 [INFO] Iteration 1: openshift-cluster-samples-operator: True
07-12 15:50:11.055 2023-07-12 19:50:10,388 [INFO] Iteration 1: openshift-console: True
07-12 15:50:11.055 2023-07-12 19:50:10,413 [INFO] Iteration 1: openshift-node: True
07-12 15:50:11.055 2023-07-12 19:50:10,445 [INFO] Iteration 1: openshift-kube-controller-manager-operator: True
07-12 15:50:11.055 2023-07-12 19:50:10,457 [INFO] Iteration 1: openshift-oauth-apiserver: True
07-12 15:50:11.055 2023-07-12 19:50:10,480 [INFO] Iteration 1: openshift-openstack-infra: True
07-12 15:50:11.055 2023-07-12 19:50:10,506 [INFO] Iteration 1: openshift-config-managed: True
07-12 15:50:11.055 2023-07-12 19:50:10,649 [INFO] Iteration 1: openshift-operator-lifecycle-manager: True
07-12 15:50:11.055 2023-07-12 19:50:10,742 [INFO] Iteration 1: openshift-dns-operator: True
07-12 15:50:11.055 2023-07-12 19:50:10,766 [INFO] Iteration 1: kube-node-lease: True
07-12 15:50:11.055 2023-07-12 19:50:10,865 [INFO] Iteration 1: default: True
07-12 15:50:11.643 2023-07-12 19:50:11,353 [INFO] Iteration 1: openshift-machine-config-operator: True
07-12 15:50:11.643 2023-07-12 19:50:11,549 [INFO] Iteration 1: openshift-kube-apiserver: True
07-12 15:50:11.940 2023-07-12 19:50:11,643 [INFO] Iteration 1: openshift-console-operator: True
07-12 15:50:11.940 2023-07-12 19:50:11,646 [INFO] Iteration 1: openshift-image-registry: True
07-12 15:50:11.940 2023-07-12 19:50:11,679 [INFO] Iteration 1: openshift-vsphere-infra: True
07-12 15:50:11.940 2023-07-12 19:50:11,742 [INFO] Iteration 1: openshift-kube-scheduler-operator: True
07-12 15:50:11.940 2023-07-12 19:50:11,767 [INFO] Iteration 1: kube-public: True
07-12 15:50:11.940 2023-07-12 19:50:11,857 [INFO] Iteration 1: openshift-marketplace: True
07-12 15:50:12.195 2023-07-12 19:50:11,962 [INFO] Iteration 1: openshift-apiserver: True
07-12 15:50:12.195 2023-07-12 19:50:12,064 [INFO] Iteration 1: openshift: True
07-12 15:50:12.195 2023-07-12 19:50:12,069 [INFO] Iteration 1: openshift-monitoring: True
07-12 15:50:12.195 2023-07-12 19:50:12,142 [INFO] Iteration 1: openshift-ingress-operator: True
07-12 15:50:12.195 2023-07-12 19:50:12,171 [INFO] Iteration 1: openshift-config: True
07-12 15:50:12.491 2023-07-12 19:50:12,249 [INFO] Iteration 1: openshift-authentication: True
07-12 15:50:12.491 2023-07-12 19:50:12,251 [INFO] Iteration 1: openshift-controller-manager: True
07-12 15:50:12.491 2023-07-12 19:50:12,273 [INFO] Iteration 1: kube-system: True
07-12 15:50:12.491 2023-07-12 19:50:12,342 [INFO] Iteration 1: openshift-etcd-operator: True
07-12 15:50:12.839 2023-07-12 19:50:12,543 [INFO] Iteration 1: openshift-kube-controller-manager: True
07-12 15:50:12.839 2023-07-12 19:50:12,638 [INFO] Iteration 1: openshift-operators: True
07-12 15:50:12.839 2023-07-12 19:50:12,646 [INFO] Iteration 1: openshift-etcd: True
07-12 15:50:12.839 2023-07-12 19:50:12,741 [INFO] Iteration 1: openshift-kube-storage-version-migrator-operator: True
07-12 15:50:12.839 2023-07-12 19:50:12,742 [INFO] Iteration 1: openshift-network-operator: True
07-12 15:50:12.839 2023-07-12 19:50:12,775 [INFO] Iteration 1: openshift-authentication-operator: True
07-12 15:50:13.094 2023-07-12 19:50:12,841 [INFO] Iteration 1: openshift-service-ca: True
07-12 15:50:13.094 2023-07-12 19:50:12,865 [INFO] Iteration 1: openshift-controller-manager-operator: True
07-12 15:50:13.094 2023-07-12 19:50:12,875 [INFO] Iteration 1: openshift-cluster-storage-operator: True
07-12 15:50:13.094 2023-07-12 19:50:12,943 [INFO] Iteration 1: openshift-cluster-node-tuning-operator: True
07-12 15:50:13.094 2023-07-12 19:50:12,950 [INFO] Iteration 1: openshift-config-operator: True
07-12 15:50:13.094 2023-07-12 19:50:12,984 [INFO] Iteration 1: openshift-multus: True
07-12 15:50:14.040 2023-07-12 19:50:13,858 [INFO] Iteration 1: openshift-cluster-csi-drivers: True
07-12 15:50:14.294 2023-07-12 19:50:14,046 [INFO] Iteration 1: openshift-dns: True
07-12 15:50:14.853 2023-07-12 19:50:14,639 [INFO] Iteration 1: openshift-ovn-kubernetes: True
07-12 15:50:14.853 2023-07-12 19:50:14,735 [INFO] Iteration 1: openshift-kube-scheduler: True
07-12 15:50:14.853 2023-07-12 19:50:14,737 [INFO] HTTP requests served: 0
07-12 15:50:14.853
07-12 15:50:15.411 2023-07-12 19:50:15,193 [INFO] []
07-12 15:50:15.411
07-12 15:50:15.411 2023-07-12 19:50:15,193 [INFO] Sleeping for the specified duration: 3
% oc get nodes
NAME STATUS ROLES AGE VERSION
........
ip-10-0-214-151.us-east-2.compute.internal Ready worker 4h41m v1.19.16+8203b20
ip-10-0-215-198.us-east-2.compute.internal Ready worker 4h41m v1.19.16+8203b20
ip-10-0-215-4.us-east-2.compute.internal Ready,SchedulingDisabled worker 4h41m v1.19.16+8203b20
ip-10-0-218-208.us-east-2.compute.internal Ready worker 4h41m v1.19.16+8203b20
ip-10-0-220-10.us-east-2.compute.internal Ready worker 4h41m v1.19.16+8203b20
ip-10-0-221-107.us-east-2.compute.internal Ready,SchedulingDisabled worker 5h1m v1.19.16+8203b20
ip-10-0-221-75.us-east-2.compute.internal Ready master 5h12m v1.19.16+8203b20
