scylla-operator
scylla-operator copied to clipboard
Published
20 hours ago •
scylladb
scylladb
Flake: ScyllaCluster Orphaned PV [It] should replace a node with orphaned PV
https://github.com/scylladb/scylla-operator/actions/runs/3067025626/jobs/4983951687#step:12:943 https://github.com/scylladb/scylla-operator/suites/8323775595/artifacts/368520674
• [FAILED] [2244.113 seconds]
ScyllaCluster Orphaned PV
github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster/scyllacluster_pv.go:22
[It] should replace a node with orphaned PV
github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster/scyllacluster_pv.go:27
Begin Captured StdOut/StdErr Output >>
I0919 15:16:40.[945](https://github.com/scylladb/scylla-operator/actions/runs/3067025626/jobs/4983951687#step:12:946)782 28 framework/framework.go:249] "Namespace removed." Namespace="e2e-test-scyllacluster-msh9g-z92kw"
<< End Captured StdOut/StdErr Output
Begin Captured GinkgoWriter Output >>
[BeforeEach] ScyllaCluster Orphaned PV
github.com/scylladb/scylla-operator/test/e2e/framework/framework.go:57
STEP: Creating a new namespace 09/19/22 14:39:16.833
Sep 19 14:39:16.900: INFO: Created namespace "e2e-test-scyllacluster-msh9g-z92kw".
STEP: Waiting for ServiceAccount "e2e-user" in namespace "e2e-test-scyllacluster-msh9g-z92kw". 09/19/22 14:39:16.935
STEP: Waiting for default ServiceAccount in namespace "e2e-test-scyllacluster-msh9g-z92kw". 09/19/22 14:39:16.994
[It] should replace a node with orphaned PV
github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster/scyllacluster_pv.go:27
STEP: Creating a ScyllaCluster 09/19/22 14:39:17.027
STEP: Waiting for the ScyllaCluster to deploy 09/19/22 14:39:17.039
Sep 19 14:46:52.764: INFO: ScyllaCluster e2e-test-scyllacluster-msh9g-z92kw/basic-xnk8b (RV=7462) is rolled out
STEP: Inserting data with RF=3 09/19/22 14:46:57.798
STEP: Verifying the ScyllaCluster 09/19/22 14:46:59.814
Sep 19 14:46:59.824: INFO: Found 3 pvc(s) in namespace "e2e-test-scyllacluster-msh9g-z92kw"
Sep 19 14:46:59.824: INFO: Found 3 pvc(s) for ScyllaCluster "e2e-test-scyllacluster-msh9g-z92kw/basic-xnk8b"
STEP: Reading data with RF=3 09/19/22 14:46:59.851
STEP: Verifying data consistency 09/19/22 14:46:59.859
STEP: Simulating a PV on node that's gone 09/19/22 14:46:59.859
STEP: Waiting for the PVC to be replaced 09/19/22 14:46:59.895
STEP: Waiting for the ScyllaCluster to observe the degradation 09/19/22 14:47:18.217
Sep 19 14:47:18.220: INFO: ScyllaCluster e2e-test-scyllacluster-msh9g-z92kw/basic-xnk8b (RV=7503) is rolled out
STEP: Waiting for the ScyllaCluster to deploy 09/19/22 14:47:23.221
[AfterEach] ScyllaCluster Orphaned PV
github.com/scylladb/scylla-operator/test/e2e/framework/framework.go:58
STEP: Collecting events from namespace "e2e-test-scyllacluster-msh9g-z92kw". 09/19/22 15:10:23.224
STEP: Found 98 events. 09/19/22 15:10:23.239
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:39:53 +0000 UTC - event for basic-xnk8b-auth-token: {scyllacluster-controller } SecretCreated: Secret e2e-test-scyllacluster-msh9g-z92kw/basic-xnk8b-auth-token created
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:39:53 +0000 UTC - event for basic-xnk8b-member: {scyllacluster-controller } ServiceAccountCreated: ServiceAccount e2e-test-scyllacluster-msh9g-z92kw/basic-xnk8b-member created
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:39:53 +0000 UTC - event for basic-xnk8b-member: {scyllacluster-controller } RoleBindingCreated: RoleBinding e2e-test-scyllacluster-msh9g-z92kw/basic-xnk8b-member created
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:39:53 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a: {statefulset-controller } SuccessfulCreate: create Claim data-basic-xnk8b-us-east-1-us-east-1a-0 Pod basic-xnk8b-us-east-1-us-east-1a-0 in StatefulSet basic-xnk8b-us-east-1-us-east-1a success
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:39:53 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a: {scyllacluster-controller } StatefulSetCreated: StatefulSet e2e-test-scyllacluster-msh9g-z92kw/basic-xnk8b-us-east-1-us-east-1a created
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:39:53 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a: {statefulset-controller } SuccessfulCreate: create Pod basic-xnk8b-us-east-1-us-east-1a-0 in StatefulSet basic-xnk8b-us-east-1-us-east-1a successful
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:39:53 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-0: {default-scheduler } FailedScheduling: 0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:39:53 +0000 UTC - event for data-basic-xnk8b-us-east-1-us-east-1a-0: {k8s.io/minikube-hostpath_fv-az348-166_bee32817-5660-4da9-8455-7de95eba2e20 } ProvisioningSucceeded: Successfully provisioned volume pvc-6d2415a3-02b5-4a69-8cc1-4e17f358573d
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:39:53 +0000 UTC - event for data-basic-xnk8b-us-east-1-us-east-1a-0: {k8s.io/minikube-hostpath_fv-az348-166_bee32817-5660-4da9-8455-7de95eba2e20 } Provisioning: External provisioner is provisioning volume for claim "e2e-test-scyllacluster-msh9g-z92kw/data-basic-xnk8b-us-east-1-us-east-1a-0"
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:39:53 +0000 UTC - event for data-basic-xnk8b-us-east-1-us-east-1a-0: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "k8s.io/minikube-hostpath" or manually created by system administrator
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:39:55 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-0: {default-scheduler } Scheduled: Successfully assigned e2e-test-scyllacluster-msh9g-z92kw/basic-xnk8b-us-east-1-us-east-1a-0 to fv-az348-166
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:40:03 +0000 UTC - event for basic-xnk8b: {scyllacluster-controller } PodDisruptionBudgetCreated: PodDisruptionBudget e2e-test-scyllacluster-msh9g-z92kw/basic-xnk8b created
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:40:03 +0000 UTC - event for basic-xnk8b-client: {scyllacluster-controller } ServiceCreated: Service e2e-test-scyllacluster-msh9g-z92kw/basic-xnk8b-client created
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:40:03 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-0: {scyllacluster-controller } ServiceCreated: Service e2e-test-scyllacluster-msh9g-z92kw/basic-xnk8b-us-east-1-us-east-1a-0 created
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:40:03 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-1: {scyllacluster-controller } ServiceCreated: Service e2e-test-scyllacluster-msh9g-z92kw/basic-xnk8b-us-east-1-us-east-1a-1 created
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:40:03 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-2: {scyllacluster-controller } ServiceCreated: Service e2e-test-scyllacluster-msh9g-z92kw/basic-xnk8b-us-east-1-us-east-1a-2 created
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:40:10 +0000 UTC - event for nodeconfig-podinfo-ac8a4294-5532-4e86-ad36-b17452b8a38a: {NodeConfigCM-controller } ConfigMapCreated: ConfigMap e2e-test-scyllacluster-msh9g-z92kw/nodeconfig-podinfo-ac8a4294-5532-4e86-ad36-b17452b8a38a created
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:40:13 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-0: {kubelet fv-az348-166} Pulled: Container image "docker.io/scylladb/scylla-operator:ci" already present on machine
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:40:13 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-0: {kubelet fv-az348-166} Started: Started container sidecar-injection
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:40:13 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-0: {kubelet fv-az348-166} Created: Created container sidecar-injection
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:40:21 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-0: {kubelet fv-az348-166} Pulled: Container image "docker.io/scylladb/scylla-manager-agent:2.6.3" already present on machine
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:40:21 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-0: {kubelet fv-az348-166} Started: Started container scylla
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:40:21 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-0: {kubelet fv-az348-166} Created: Created container scylla
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:40:21 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-0: {kubelet fv-az348-166} Pulled: Container image "docker.io/scylladb/scylla:4.6.3" already present on machine
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:40:22 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-0: {kubelet fv-az348-166} Started: Started container scylla-manager-agent
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:40:22 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-0: {kubelet fv-az348-166} Created: Created container scylla-manager-agent
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:40:25 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-0: {kubelet fv-az348-166} Unhealthy: Startup probe failed: Get "http://172.17.0.26:8080/healthz": dial tcp 172.17.0.26:8080: connect: connection refused
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:40:30 +0000 UTC - event for nodeconfig-podinfo-ac8a4294-5532-4e86-ad36-b17452b8a38a: {NodeConfigCM-controller } ConfigMapUpdated: ConfigMap e2e-test-scyllacluster-msh9g-z92kw/nodeconfig-podinfo-ac8a4294-5532-4e86-ad36-b17452b8a38a updated
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:40:45 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-0: {kubelet fv-az348-166} Unhealthy: Startup probe failed: HTTP probe failed with statuscode: 503
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:41:05 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-0: {kubelet fv-az348-166} Unhealthy: Readiness probe failed: HTTP probe failed with statuscode: 503
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:41:49 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a: {statefulset-controller } SuccessfulCreate: create Pod basic-xnk8b-us-east-1-us-east-1a-1 in StatefulSet basic-xnk8b-us-east-1-us-east-1a successful
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:41:49 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a: {statefulset-controller } SuccessfulCreate: create Claim data-basic-xnk8b-us-east-1-us-east-1a-1 Pod basic-xnk8b-us-east-1-us-east-1a-1 in StatefulSet basic-xnk8b-us-east-1-us-east-1a success
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:41:49 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-1: {default-scheduler } FailedScheduling: 0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:41:49 +0000 UTC - event for data-basic-xnk8b-us-east-1-us-east-1a-1: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "k8s.io/minikube-hostpath" or manually created by system administrator
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:41:49 +0000 UTC - event for data-basic-xnk8b-us-east-1-us-east-1a-1: {k8s.io/minikube-hostpath_fv-az348-166_bee32817-5660-4da9-8455-7de95eba2e20 } Provisioning: External provisioner is provisioning volume for claim "e2e-test-scyllacluster-msh9g-z92kw/data-basic-xnk8b-us-east-1-us-east-1a-1"
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:41:50 +0000 UTC - event for data-basic-xnk8b-us-east-1-us-east-1a-1: {k8s.io/minikube-hostpath_fv-az348-166_bee32817-5660-4da9-8455-7de95eba2e20 } ProvisioningSucceeded: Successfully provisioned volume pvc-cdfb51f0-3ee2-4ac6-9760-a79d[947](https://github.com/scylladb/scylla-operator/actions/runs/3067025626/jobs/4983951687#step:12:948)bca74
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:41:51 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-1: {default-scheduler } Scheduled: Successfully assigned e2e-test-scyllacluster-msh9g-z92kw/basic-xnk8b-us-east-1-us-east-1a-1 to fv-az348-166
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:41:57 +0000 UTC - event for nodeconfig-podinfo-df3176ae-55e9-4da4-bcc5-82e69d93ecd7: {NodeConfigCM-controller } ConfigMapCreated: ConfigMap e2e-test-scyllacluster-msh9g-z92kw/nodeconfig-podinfo-df3176ae-55e9-4da4-bcc5-82e69d93ecd7 created
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:42:05 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-1: {kubelet fv-az348-166} Started: Started container sidecar-injection
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:42:05 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-1: {kubelet fv-az348-166} Created: Created container sidecar-injection
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:42:05 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-1: {kubelet fv-az348-166} Pulled: Container image "docker.io/scylladb/scylla-operator:ci" already present on machine
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:42:14 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-1: {kubelet fv-az348-166} Created: Created container scylla-manager-agent
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:42:14 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-1: {kubelet fv-az348-166} Pulled: Container image "docker.io/scylladb/scylla:4.6.3" already present on machine
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:42:14 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-1: {kubelet fv-az348-166} Created: Created container scylla
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:42:14 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-1: {kubelet fv-az348-166} Started: Started container scylla
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:42:14 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-1: {kubelet fv-az348-166} Pulled: Container image "docker.io/scylladb/scylla-manager-agent:2.6.3" already present on machine
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:42:15 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-1: {kubelet fv-az348-166} Started: Started container scylla-manager-agent
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:42:19 +0000 UTC - event for nodeconfig-podinfo-df3176ae-55e9-4da4-bcc5-82e69d93ecd7: {NodeConfigCM-controller } ConfigMapUpdated: ConfigMap e2e-test-scyllacluster-msh9g-z92kw/nodeconfig-podinfo-df3176ae-55e9-4da4-bcc5-82e69d93ecd7 updated
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:42:21 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-1: {kubelet fv-az348-166} Unhealthy: Startup probe failed: Get "http://172.17.0.39:8080/healthz": dial tcp 172.17.0.39:8080: connect: connection refused
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:42:31 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-1: {kubelet fv-az348-166} Unhealthy: Startup probe failed: HTTP probe failed with statuscode: 503
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:42:41 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-1: {kubelet fv-az348-166} Unhealthy: Readiness probe failed: HTTP probe failed with statuscode: 500
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:42:51 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-1: {kubelet fv-az348-166} Unhealthy: Readiness probe failed: HTTP probe failed with statuscode: 503
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:44:31 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a: {statefulset-controller } SuccessfulCreate: create Claim data-basic-xnk8b-us-east-1-us-east-1a-2 Pod basic-xnk8b-us-east-1-us-east-1a-2 in StatefulSet basic-xnk8b-us-east-1-us-east-1a success
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:44:31 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a: {statefulset-controller } SuccessfulCreate: create Pod basic-xnk8b-us-east-1-us-east-1a-2 in StatefulSet basic-xnk8b-us-east-1-us-east-1a successful
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:44:31 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-2: {default-scheduler } FailedScheduling: 0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:44:31 +0000 UTC - event for data-basic-xnk8b-us-east-1-us-east-1a-2: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "k8s.io/minikube-hostpath" or manually created by system administrator
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:44:31 +0000 UTC - event for data-basic-xnk8b-us-east-1-us-east-1a-2: {k8s.io/minikube-hostpath_fv-az348-166_61c62abc-e272-4d75-ad8d-3b220119ab42 } Provisioning: External provisioner is provisioning volume for claim "e2e-test-scyllacluster-msh9g-z92kw/data-basic-xnk8b-us-east-1-us-east-1a-2"
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:44:31 +0000 UTC - event for data-basic-xnk8b-us-east-1-us-east-1a-2: {k8s.io/minikube-hostpath_fv-az348-166_61c62abc-e272-4d75-ad8d-3b220119ab42 } ProvisioningSucceeded: Successfully provisioned volume pvc-9ee9d9cc-6061-473d-9bb6-c99ae0b8b33d
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:44:32 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-2: {default-scheduler } Scheduled: Successfully assigned e2e-test-scyllacluster-msh9g-z92kw/basic-xnk8b-us-east-1-us-east-1a-2 to fv-az348-166
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:44:32 +0000 UTC - event for nodeconfig-podinfo-cf1fe30f-12df-4b85-8c30-5b891517ac0d: {NodeConfigCM-controller } ConfigMapCreated: ConfigMap e2e-test-scyllacluster-msh9g-z92kw/nodeconfig-podinfo-cf1fe30f-12df-4b85-8c30-5b891517ac0d created
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:44:35 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-2: {kubelet fv-az348-166} Pulled: Container image "docker.io/scylladb/scylla-operator:ci" already present on machine
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:44:35 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-2: {kubelet fv-az348-166} Created: Created container sidecar-injection
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:44:36 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-2: {kubelet fv-az348-166} Started: Started container sidecar-injection
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:44:44 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-2: {kubelet fv-az348-166} Pulled: Container image "docker.io/scylladb/scylla:4.6.3" already present on machine
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:44:44 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-2: {kubelet fv-az348-166} Created: Created container scylla
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:44:45 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-2: {kubelet fv-az348-166} Pulled: Container image "docker.io/scylladb/scylla-manager-agent:2.6.3" already present on machine
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:44:45 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-2: {kubelet fv-az348-166} Started: Started container scylla-manager-agent
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:44:45 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-2: {kubelet fv-az348-166} Created: Created container scylla-manager-agent
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:44:45 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-2: {kubelet fv-az348-166} Started: Started container scylla
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:44:49 +0000 UTC - event for nodeconfig-podinfo-cf1fe30f-12df-4b85-8c30-5b891517ac0d: {NodeConfigCM-controller } ConfigMapUpdated: ConfigMap e2e-test-scyllacluster-msh9g-z92kw/nodeconfig-podinfo-cf1fe30f-12df-4b85-8c30-5b891517ac0d updated
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:44:52 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-2: {kubelet fv-az348-166} Unhealthy: Startup probe failed: Get "http://172.17.0.33:8080/healthz": dial tcp 172.17.0.33:8080: connect: connection refused
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:45:03 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-2: {kubelet fv-az348-166} Unhealthy: Readiness probe failed: HTTP probe failed with statuscode: 503
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:46:59 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-2: {scyllacluster-controller } ServiceUpdated: Service e2e-test-scyllacluster-msh9g-z92kw/basic-xnk8b-us-east-1-us-east-1a-2 updated
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:46:59 +0000 UTC - event for data-basic-xnk8b-us-east-1-us-east-1a-2: {scyllacluster-controller } PersistentVolumeClaimDeleted: PersistentVolumeClaim e2e-test-scyllacluster-msh9g-z92kw/data-basic-xnk8b-us-east-1-us-east-1a-2 deleted
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:47:09 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-2: {kubelet fv-az348-166} Killing: Stopping container scylla
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:47:09 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-2: {kubelet fv-az348-166} Killing: Stopping container scylla-manager-agent
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:47:09 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-2: {scyllacluster-controller } PersistentVolumeClaimDeleted: PersistentVolumeClaim e2e-test-scyllacluster-msh9g-z92kw/basic-xnk8b-us-east-1-us-east-1a-2 deleted
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:47:10 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-2: {scyllacluster-controller } ServiceDeleted: Service e2e-test-scyllacluster-msh9g-z92kw/basic-xnk8b-us-east-1-us-east-1a-2 deleted
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:47:18 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a: {statefulset-controller } FailedCreate: create Pod basic-xnk8b-us-east-1-us-east-1a-2 in StatefulSet basic-xnk8b-us-east-1-us-east-1a failed error: pvc data-basic-xnk8b-us-east-1-us-east-1a-2 is being deleted
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:47:18 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-2: {default-scheduler } FailedScheduling: 0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:47:18 +0000 UTC - event for data-basic-xnk8b-us-east-1-us-east-1a-2: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "k8s.io/minikube-hostpath" or manually created by system administrator
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:47:18 +0000 UTC - event for data-basic-xnk8b-us-east-1-us-east-1a-2: {k8s.io/minikube-hostpath_fv-az348-166_61c62abc-e272-4d75-ad8d-3b220119ab42 } ProvisioningSucceeded: Successfully provisioned volume pvc-f482c3c9-f0c2-4594-84d2-72fde0e2ab94
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:47:18 +0000 UTC - event for data-basic-xnk8b-us-east-1-us-east-1a-2: {k8s.io/minikube-hostpath_fv-az348-166_61c62abc-e272-4d75-ad8d-3b220119ab42 } Provisioning: External provisioner is provisioning volume for claim "e2e-test-scyllacluster-msh9g-z92kw/data-basic-xnk8b-us-east-1-us-east-1a-2"
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:47:19 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-2: {default-scheduler } Scheduled: Successfully assigned e2e-test-scyllacluster-msh9g-z92kw/basic-xnk8b-us-east-1-us-east-1a-2 to fv-az348-166
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:47:19 +0000 UTC - event for nodeconfig-podinfo-707eba1e-2ba1-4860-a83c-b5eb647e7e92: {NodeConfigCM-controller } ConfigMapCreated: ConfigMap e2e-test-scyllacluster-msh9g-z92kw/nodeconfig-podinfo-707eba1e-2ba1-4860-a83c-b5eb647e7e92 created
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:47:20 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-2: {kubelet fv-az348-166} Pulled: Container image "docker.io/scylladb/scylla-operator:ci" already present on machine
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:47:20 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-2: {kubelet fv-az348-166} Created: Created container sidecar-injection
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:47:21 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-2: {kubelet fv-az348-166} Started: Started container sidecar-injection
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:47:28 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-2: {kubelet fv-az348-166} Created: Created container scylla
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:47:28 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-2: {kubelet fv-az348-166} Started: Started container scylla
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:47:28 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-2: {kubelet fv-az348-166} Pulled: Container image "docker.io/scylladb/scylla-manager-agent:2.6.3" already present on machine
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:47:28 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-2: {kubelet fv-az348-166} Pulled: Container image "docker.io/scylladb/scylla:4.6.3" already present on machine
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:47:29 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-2: {kubelet fv-az348-166} Created: Created container scylla-manager-agent
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:47:29 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-2: {kubelet fv-az348-166} Started: Started container scylla-manager-agent
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:47:29 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-2: {kubelet fv-az348-166} Unhealthy: Startup probe failed: Get "http://172.17.0.22:8080/healthz": dial tcp 172.17.0.22:8080: connect: connection refused
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:47:30 +0000 UTC - event for nodeconfig-podinfo-707eba1e-2ba1-4860-a83c-b5eb647e7e92: {NodeConfigCM-controller } ConfigMapUpdated: ConfigMap e2e-test-scyllacluster-msh9g-z92kw/nodeconfig-podinfo-707eba1e-2ba1-4860-a83c-b5eb647e7e92 updated
Sep 19 15:10:23.239: INFO: At 2022-09-19 14:47:39 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-2: {kubelet fv-az348-166} Unhealthy: Readiness probe failed: HTTP probe failed with statuscode: 503
Sep 19 15:10:23.239: INFO: At 2022-09-19 15:02:29 +0000 UTC - event for basic-xnk8b-us-east-1-us-east-1a-2: {kubelet fv-az348-166} Unhealthy: Liveness probe failed: HTTP probe failed with statuscode: 503
STEP: Collecting dumps from namespace "e2e-test-scyllacluster-msh9g-z92kw". 09/19/22 15:10:23.239
STEP: Destroying namespace "e2e-test-scyllacluster-msh9g-z92kw". 09/19/22 15:10:23.659
STEP: Waiting for namespace "e2e-test-scyllacluster-msh9g-z92kw" to be removed. 09/19/22 15:10:23.663
<< End Captured GinkgoWriter Output
Unexpected error:
<*errors.errorString | 0xc0002530b0>: {
s: "timed out waiting for the condition",
}
timed out waiting for the condition
occurred
In [It] at: github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster/scyllacluster_pv.go:123
Full Stack Trace
github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster.glob..func5.1()
github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster/scyllacluster_pv.go:123 +0x100e
@tnozicka Any chance you've got a local copy of the artifacts? I believe they were deleted when the job was re-run.
