scylla-operator icon indicating copy to clipboard operation
scylla-operator copied to clipboard

Published 20 hours ago • verified publisher icon scylladb

Flake: ScyllaCluster upgrades should deploy and update [It] with 3 member(s) and 1 rack(s) from 4.6.2 to 4.6.3

Open tnozicka opened this issue 3 years ago • 6 comments

I saw this on a PR that only touched an isolated e2e so this should be a real flake.

https://github.com/scylladb/scylla-operator/actions/runs/3066479399/jobs/4951738619#step:12:765 https://github.com/scylladb/scylla-operator/suites/8322409171/artifacts/365734281

• [FAILED] [760.877 seconds]
ScyllaCluster upgrades
github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster/scyllacluster_upgrades.go:19
  should deploy and update
  github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster/scyllacluster_upgrades.go:35
    [It] with 3 member(s) and 1 rack(s) from 4.6.2 to 4.6.3
    github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster/scyllacluster_upgrades.go:106

...

  Expected
      <string>: basic-h8jj7-us-east-1-rack-0-648f4c4bbb
  to equal
      <string>: basic-h8jj7-us-east-1-rack-0-785f5cd8f8
  In [It] at: github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster/verify.go:63

  Full Stack Trace
    github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster.verifyStatefulset(0xc0002b2a00)
    	github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster/verify.go:63 +0x292
    github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster.verifyScyllaCluster({0x1d4b648, 0xc000483680}, {0x1d62270, 0xc0002aa900}, 0xc00075a000, 0xc000089310)
    	github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster/verify.go:99 +0xc7a
    github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster.glob..func13.2(0xc000583200)
    	github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster/scyllacluster_upgrades.go:97 +0xc54
    reflect.Value.call({0x1783a00?, 0xc0003b1bb0?, 0x7f4bd040e5b8?}, {0x1aa05f0, 0x4}, {0xc000251ea8, 0x1, 0xc000627ea8?})
    	reflect/value.go:584 +0x8c5
    reflect.Value.Call({0x1783a00?, 0xc0003b1bb0?, 0x1d5f3a0?}, {0xc000251ea8?, 0x467bce?, 0x43bb27?})
    	reflect/value.go:368 +0xbc

[BeforeEach] ScyllaCluster upgrades
      github.com/scylladb/scylla-operator/test/e2e/framework/framework.go:57
    STEP: Creating a new namespace 09/16/22 08:40:22.69
    Sep 16 08:40:22.788: INFO: Created namespace "e2e-test-scyllacluster-mjgn4-6ndph".
    STEP: Waiting for ServiceAccount "e2e-user" in namespace "e2e-test-scyllacluster-mjgn4-6ndph". 09/16/22 08:40:22.835
    STEP: Waiting for default ServiceAccount in namespace "e2e-test-scyllacluster-mjgn4-6ndph". 09/16/22 08:40:24.691
    [It] with 3 member(s) and 1 rack(s) from 4.6.2 to 4.6.3
      github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster/scyllacluster_upgrades.go:106
    STEP: Creating a ScyllaCluster 09/16/22 08:40:24.696
    STEP: Waiting for the ScyllaCluster to deploy 09/16/22 08:40:24.731
    Sep 16 08:48:48.502: INFO: ScyllaCluster e2e-test-scyllacluster-mjgn4-6ndph/basic-h8jj7 (RV=7123) is rolled out
    STEP: Inserting data with RF=3 09/16/22 08:48:53.527
    STEP: Verifying the ScyllaCluster 09/16/22 08:48:55.675
    Sep 16 08:48:55.694: INFO: Found 3 pvc(s) in namespace "e2e-test-scyllacluster-mjgn4-6ndph"
    Sep 16 08:48:55.694: INFO: Found 3 pvc(s) for ScyllaCluster "e2e-test-scyllacluster-mjgn4-6ndph/basic-h8jj7"
    STEP: Reading data with RF=3 09/16/22 08:48:55.723
    STEP: Verifying data consistency 09/16/22 08:48:55.733
    STEP: triggering and update 09/16/22 08:48:55.733
    STEP: Waiting for the ScyllaCluster to re-deploy 09/16/22 08:48:55.746
    Sep 16 08:48:55.748: INFO: ScyllaCluster e2e-test-scyllacluster-mjgn4-6ndph/basic-h8jj7 (RV=7123) is rolled out
    STEP: Verifying the ScyllaCluster 09/16/22 08:49:00.826
    [AfterEach] ScyllaCluster upgrades
      github.com/scylladb/scylla-operator/test/e2e/framework/framework.go:58
    STEP: Collecting events from namespace "e2e-test-scyllacluster-mjgn4-6ndph". 09/16/22 08:49:00.834
    STEP: Found 77 events. 09/16/22 08:49:00.874
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:40:43 +0000 UTC - event for basic-h8jj7-auth-token: {scyllacluster-controller } SecretCreated: Secret e2e-test-scyllacluster-mjgn4-6ndph/basic-h8jj7-auth-token created
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:40:43 +0000 UTC - event for basic-h8jj7-member: {scyllacluster-controller } ServiceAccountCreated: ServiceAccount e2e-test-scyllacluster-mjgn4-6ndph/basic-h8jj7-member created
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:40:43 +0000 UTC - event for basic-h8jj7-member: {scyllacluster-controller } RoleBindingCreated: RoleBinding e2e-test-scyllacluster-mjgn4-6ndph/basic-h8jj7-member created
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:40:43 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0: {statefulset-controller } SuccessfulCreate: create Claim data-basic-h8jj7-us-east-1-rack-0-0 Pod basic-h8jj7-us-east-1-rack-0-0 in StatefulSet basic-h8jj7-us-east-1-rack-0 success
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:40:43 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0: {scyllacluster-controller } StatefulSetCreated: StatefulSet e2e-test-scyllacluster-mjgn4-6ndph/basic-h8jj7-us-east-1-rack-0 created
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:40:43 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0: {statefulset-controller } SuccessfulCreate: create Pod basic-h8jj7-us-east-1-rack-0-0 in StatefulSet basic-h8jj7-us-east-1-rack-0 successful
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:40:43 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-0: {default-scheduler } FailedScheduling: 0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:40:43 +0000 UTC - event for data-basic-h8jj7-us-east-1-rack-0-0: {k8s.io/minikube-hostpath_fv-az75-556_8a7df3ef-b96e-4204-b896-2402b7a93870 } ProvisioningSucceeded: Successfully provisioned volume pvc-e76fca9f-e226-41c8-bf81-9ed8b2067dce
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:40:43 +0000 UTC - event for data-basic-h8jj7-us-east-1-rack-0-0: {k8s.io/minikube-hostpath_fv-az75-556_8a7df3ef-b96e-4204-b896-2402b7a93870 } Provisioning: External provisioner is provisioning volume for claim "e2e-test-scyllacluster-mjgn4-6ndph/data-basic-h8jj7-us-east-1-rack-0-0"
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:40:43 +0000 UTC - event for data-basic-h8jj7-us-east-1-rack-0-0: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "k8s.io/minikube-hostpath" or manually created by system administrator
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:40:45 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-0: {default-scheduler } Scheduled: Successfully assigned e2e-test-scyllacluster-mjgn4-6ndph/basic-h8jj7-us-east-1-rack-0-0 to fv-az75-556
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:40:53 +0000 UTC - event for basic-h8jj7: {scyllacluster-controller } PodDisruptionBudgetCreated: PodDisruptionBudget e2e-test-scyllacluster-mjgn4-6ndph/basic-h8jj7 created
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:40:53 +0000 UTC - event for basic-h8jj7-client: {scyllacluster-controller } ServiceCreated: Service e2e-test-scyllacluster-mjgn4-6ndph/basic-h8jj7-client created
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:40:53 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-0: {scyllacluster-controller } ServiceCreated: Service e2e-test-scyllacluster-mjgn4-6ndph/basic-h8jj7-us-east-1-rack-0-0 created
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:40:53 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-1: {scyllacluster-controller } ServiceCreated: Service e2e-test-scyllacluster-mjgn4-6ndph/basic-h8jj7-us-east-1-rack-0-1 created
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:40:53 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-2: {scyllacluster-controller } ServiceCreated: Service e2e-test-scyllacluster-mjgn4-6ndph/basic-h8jj7-us-east-1-rack-0-2 created
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:41:09 +0000 UTC - event for nodeconfig-podinfo-d18759c5-d351-4c6a-9d5b-b0a6e9a7b5fb: {NodeConfigCM-controller } ConfigMapCreated: ConfigMap e2e-test-scyllacluster-mjgn4-6ndph/nodeconfig-podinfo-d18759c5-d351-4c6a-9d5b-b0a6e9a7b5fb created
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:41:10 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-0: {kubelet fv-az75-556} Created: Created container sidecar-injection
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:41:10 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-0: {kubelet fv-az75-556} Pulled: Container image "docker.io/scylladb/scylla-operator:ci" already present on machine
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:41:11 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-0: {kubelet fv-az75-556} Started: Started container sidecar-injection
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:41:19 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-0: {kubelet fv-az75-556} Pulled: Container image "docker.io/scylladb/scylla:4.6.2" already present on machine
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:41:20 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-0: {kubelet fv-az75-556} Created: Created container scylla
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:41:22 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-0: {kubelet fv-az75-556} Started: Started container scylla
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:41:22 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-0: {kubelet fv-az75-556} Pulled: Container image "docker.io/scylladb/scylla-manager-agent:2.6.3" already present on machine
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:41:22 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-0: {kubelet fv-az75-556} Created: Created container scylla-manager-agent
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:41:23 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-0: {kubelet fv-az75-556} Started: Started container scylla-manager-agent
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:41:25 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-0: {kubelet fv-az75-556} Unhealthy: Startup probe failed: Get "http://172.17.0.27:8080/healthz": dial tcp 172.17.0.27:8080: connect: connection refused
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:41:39 +0000 UTC - event for nodeconfig-podinfo-d18759c5-d351-4c6a-9d5b-b0a6e9a7b5fb: {NodeConfigCM-controller } ConfigMapUpdated: ConfigMap e2e-test-scyllacluster-mjgn4-6ndph/nodeconfig-podinfo-d18759c5-d351-4c6a-9d5b-b0a6e9a7b5fb updated
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:41:45 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-0: {kubelet fv-az75-556} Unhealthy: Startup probe failed: HTTP probe failed with statuscode: 503
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:42:05 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-0: {kubelet fv-az75-556} Unhealthy: Readiness probe failed: HTTP probe failed with statuscode: 500
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:42:25 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-0: {kubelet fv-az75-556} Unhealthy: Readiness probe failed: HTTP probe failed with statuscode: 503
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:43:10 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0: {statefulset-controller } SuccessfulCreate: create Pod basic-h8jj7-us-east-1-rack-0-1 in StatefulSet basic-h8jj7-us-east-1-rack-0 successful
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:43:10 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0: {statefulset-controller } SuccessfulCreate: create Claim data-basic-h8jj7-us-east-1-rack-0-1 Pod basic-h8jj7-us-east-1-rack-0-1 in StatefulSet basic-h8jj7-us-east-1-rack-0 success
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:43:10 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-1: {default-scheduler } FailedScheduling: 0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:43:10 +0000 UTC - event for data-basic-h8jj7-us-east-1-rack-0-1: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "k8s.io/minikube-hostpath" or manually created by system administrator
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:43:48 +0000 UTC - event for data-basic-h8jj7-us-east-1-rack-0-1: {k8s.io/minikube-hostpath_fv-az75-556_7fdc1012-cb5f-46b9-96d3-10d9d324748d } Provisioning: External provisioner is provisioning volume for claim "e2e-test-scyllacluster-mjgn4-6ndph/data-basic-h8jj7-us-east-1-rack-0-1"
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:43:49 +0000 UTC - event for data-basic-h8jj7-us-east-1-rack-0-1: {k8s.io/minikube-hostpath_fv-az75-556_7fdc1012-cb5f-46b9-96d3-10d9d324748d } ProvisioningSucceeded: Successfully provisioned volume pvc-f7c1b3f2-b040-46ab-bd8f-35399d69e8ef
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:43:56 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-1: {default-scheduler } Scheduled: Successfully assigned e2e-test-scyllacluster-mjgn4-6ndph/basic-h8jj7-us-east-1-rack-0-1 to fv-az75-556
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:43:56 +0000 UTC - event for nodeconfig-podinfo-84[660](https://github.com/scylladb/scylla-operator/actions/runs/3066479399/jobs/4951738619#step:12:661)45e-6a70-4051-ac54-f8086dcf25f3: {NodeConfigCM-controller } ConfigMapCreated: ConfigMap e2e-test-scyllacluster-mjgn4-6ndph/nodeconfig-podinfo-8466045e-6a70-4051-ac54-f8086dcf25f3 created
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:44:03 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-1: {kubelet fv-az75-556} Pulled: Container image "docker.io/scylladb/scylla-operator:ci" already present on machine
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:44:03 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-1: {kubelet fv-az75-556} Created: Created container sidecar-injection
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:44:04 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-1: {kubelet fv-az75-556} Started: Started container sidecar-injection
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:44:13 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-1: {kubelet fv-az75-556} Pulled: Container image "docker.io/scylladb/scylla-manager-agent:2.6.3" already present on machine
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:44:13 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-1: {kubelet fv-az75-556} Pulled: Container image "docker.io/scylladb/scylla:4.6.2" already present on machine
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:44:13 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-1: {kubelet fv-az75-556} Created: Created container scylla
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:44:13 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-1: {kubelet fv-az75-556} Started: Started container scylla
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:44:14 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-1: {kubelet fv-az75-556} Created: Created container scylla-manager-agent
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:44:14 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-1: {kubelet fv-az75-556} Started: Started container scylla-manager-agent
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:44:16 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-1: {kubelet fv-az75-556} Unhealthy: Startup probe failed: Get "http://172.17.0.29:8080/healthz": dial tcp 172.17.0.29:8080: connect: connection refused
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:44:19 +0000 UTC - event for nodeconfig-podinfo-8466045e-6a70-4051-ac54-f8086dcf25f3: {NodeConfigCM-controller } ConfigMapUpdated: ConfigMap e2e-test-scyllacluster-mjgn4-6ndph/nodeconfig-podinfo-8466045e-6a70-4051-ac54-f8086dcf25f3 updated
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:44:26 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-1: {kubelet fv-az75-556} Unhealthy: Startup probe failed: HTTP probe failed with statuscode: 503
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:44:36 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-1: {kubelet fv-az75-556} Unhealthy: Readiness probe failed: HTTP probe failed with statuscode: 500
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:44:46 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-1: {kubelet fv-az75-556} Unhealthy: Readiness probe failed: HTTP probe failed with statuscode: 503
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:46:26 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0: {statefulset-controller } SuccessfulCreate: create Claim data-basic-h8jj7-us-east-1-rack-0-2 Pod basic-h8jj7-us-east-1-rack-0-2 in StatefulSet basic-h8jj7-us-east-1-rack-0 success
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:46:26 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0: {statefulset-controller } SuccessfulCreate: create Pod basic-h8jj7-us-east-1-rack-0-2 in StatefulSet basic-h8jj7-us-east-1-rack-0 successful
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:46:26 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-2: {default-scheduler } FailedScheduling: 0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:46:26 +0000 UTC - event for data-basic-h8jj7-us-east-1-rack-0-2: {k8s.io/minikube-hostpath_fv-az75-556_7fdc1012-cb5f-46b9-96d3-10d9d324748d } ProvisioningSucceeded: Successfully provisioned volume pvc-a2509279-0da9-4bc6-9974-77953db256ab
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:46:26 +0000 UTC - event for data-basic-h8jj7-us-east-1-rack-0-2: {k8s.io/minikube-hostpath_fv-az75-556_7fdc1012-cb5f-46b9-96d3-10d9d324748d } Provisioning: External provisioner is provisioning volume for claim "e2e-test-scyllacluster-mjgn4-6ndph/data-basic-h8jj7-us-east-1-rack-0-2"
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:46:26 +0000 UTC - event for data-basic-h8jj7-us-east-1-rack-0-2: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "k8s.io/minikube-hostpath" or manually created by system administrator
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:46:28 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-2: {default-scheduler } Scheduled: Successfully assigned e2e-test-scyllacluster-mjgn4-6ndph/basic-h8jj7-us-east-1-rack-0-2 to fv-az75-556
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:46:29 +0000 UTC - event for nodeconfig-podinfo-a409a6bc-64b2-4e5f-98d6-071c9c82a87e: {NodeConfigCM-controller } ConfigMapCreated: ConfigMap e2e-test-scyllacluster-mjgn4-6ndph/nodeconfig-podinfo-a409a6bc-64b2-4e5f-98d6-071c9c82a87e created
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:46:30 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-2: {kubelet fv-az75-556} Pulled: Container image "docker.io/scylladb/scylla-operator:ci" already present on machine
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:46:30 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-2: {kubelet fv-az75-556} Created: Created container sidecar-injection
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:46:31 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-2: {kubelet fv-az75-556} Started: Started container sidecar-injection
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:46:40 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-2: {kubelet fv-az75-556} Pulled: Container image "docker.io/scylladb/scylla:4.6.2" already present on machine
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:46:40 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-2: {kubelet fv-az75-556} Created: Created container scylla
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:46:41 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-2: {kubelet fv-az75-556} Started: Started container scylla
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:46:41 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-2: {kubelet fv-az75-556} Pulled: Container image "docker.io/scylladb/scylla-manager-agent:2.6.3" already present on machine
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:46:42 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-2: {kubelet fv-az75-556} Created: Created container scylla-manager-agent
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:46:42 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-2: {kubelet fv-az75-556} Started: Started container scylla-manager-agent
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:46:43 +0000 UTC - event for nodeconfig-podinfo-a409a6bc-64b2-4e5f-98d6-071c9c82a87e: {NodeConfigCM-controller } ConfigMapUpdated: ConfigMap e2e-test-scyllacluster-mjgn4-6ndph/nodeconfig-podinfo-a409a6bc-64b2-4e5f-98d6-071c9c82a87e updated
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:46:48 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-2: {kubelet fv-az75-556} Unhealthy: Startup probe failed: HTTP probe failed with statuscode: 503
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:46:58 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-2: {kubelet fv-az75-556} Unhealthy: Readiness probe failed: HTTP probe failed with statuscode: 503
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:48:55 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0: {statefulset-controller } SuccessfulDelete: delete Pod basic-h8jj7-us-east-1-rack-0-2 in StatefulSet basic-h8jj7-us-east-1-rack-0 successful
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:48:55 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0: {scyllacluster-controller } StatefulSetUpdated: StatefulSet e2e-test-scyllacluster-mjgn4-6ndph/basic-h8jj7-us-east-1-rack-0 updated
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:48:55 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-2: {kubelet fv-az75-556} Killing: Stopping container scylla-manager-agent
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:48:55 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-2: {kubelet fv-az75-556} Killing: Stopping container scylla
    STEP: Collecting dumps from namespace "e2e-test-scyllacluster-mjgn4-6ndph". 09/16/22 08:49:00.875
    STEP: Destroying namespace "e2e-test-scyllacluster-mjgn4-6ndph". 09/16/22 08:49:01.473
    STEP: Waiting for namespace "e2e-test-scyllacluster-mjgn4-6ndph" to be removed. 09/16/22 08:49:01.479
  << End Captured GinkgoWriter Output[BeforeEach] ScyllaCluster upgrades
      github.com/scylladb/scylla-operator/test/e2e/framework/framework.go:57
    STEP: Creating a new namespace 09/16/22 08:40:22.69
    Sep 16 08:40:22.788: INFO: Created namespace "e2e-test-scyllacluster-mjgn4-6ndph".
    STEP: Waiting for ServiceAccount "e2e-user" in namespace "e2e-test-scyllacluster-mjgn4-6ndph". 09/16/22 08:40:22.835
    STEP: Waiting for default ServiceAccount in namespace "e2e-test-scyllacluster-mjgn4-6ndph". 09/16/22 08:40:24.691
    [It] with 3 member(s) and 1 rack(s) from 4.6.2 to 4.6.3
      github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster/scyllacluster_upgrades.go:106
    STEP: Creating a ScyllaCluster 09/16/22 08:40:24.696
    STEP: Waiting for the ScyllaCluster to deploy 09/16/22 08:40:24.731
    Sep 16 08:48:48.502: INFO: ScyllaCluster e2e-test-scyllacluster-mjgn4-6ndph/basic-h8jj7 (RV=7123) is rolled out
    STEP: Inserting data with RF=3 09/16/22 08:48:53.527
    STEP: Verifying the ScyllaCluster 09/16/22 08:48:55.675
    Sep 16 08:48:55.694: INFO: Found 3 pvc(s) in namespace "e2e-test-scyllacluster-mjgn4-6ndph"
    Sep 16 08:48:55.694: INFO: Found 3 pvc(s) for ScyllaCluster "e2e-test-scyllacluster-mjgn4-6ndph/basic-h8jj7"
    STEP: Reading data with RF=3 09/16/22 08:48:55.723
    STEP: Verifying data consistency 09/16/22 08:48:55.733
    STEP: triggering and update 09/16/22 08:48:55.733
    STEP: Waiting for the ScyllaCluster to re-deploy 09/16/22 08:48:55.746
    Sep 16 08:48:55.748: INFO: ScyllaCluster e2e-test-scyllacluster-mjgn4-6ndph/basic-h8jj7 (RV=7123) is rolled out
    STEP: Verifying the ScyllaCluster 09/16/22 08:49:00.826
    [AfterEach] ScyllaCluster upgrades
      github.com/scylladb/scylla-operator/test/e2e/framework/framework.go:58
    STEP: Collecting events from namespace "e2e-test-scyllacluster-mjgn4-6ndph". 09/16/22 08:49:00.834
    STEP: Found 77 events. 09/16/22 08:49:00.874
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:40:43 +0000 UTC - event for basic-h8jj7-auth-token: {scyllacluster-controller } SecretCreated: Secret e2e-test-scyllacluster-mjgn4-6ndph/basic-h8jj7-auth-token created
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:40:43 +0000 UTC - event for basic-h8jj7-member: {scyllacluster-controller } ServiceAccountCreated: ServiceAccount e2e-test-scyllacluster-mjgn4-6ndph/basic-h8jj7-member created
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:40:43 +0000 UTC - event for basic-h8jj7-member: {scyllacluster-controller } RoleBindingCreated: RoleBinding e2e-test-scyllacluster-mjgn4-6ndph/basic-h8jj7-member created
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:40:43 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0: {statefulset-controller } SuccessfulCreate: create Claim data-basic-h8jj7-us-east-1-rack-0-0 Pod basic-h8jj7-us-east-1-rack-0-0 in StatefulSet basic-h8jj7-us-east-1-rack-0 success
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:40:43 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0: {scyllacluster-controller } StatefulSetCreated: StatefulSet e2e-test-scyllacluster-mjgn4-6ndph/basic-h8jj7-us-east-1-rack-0 created
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:40:43 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0: {statefulset-controller } SuccessfulCreate: create Pod basic-h8jj7-us-east-1-rack-0-0 in StatefulSet basic-h8jj7-us-east-1-rack-0 successful
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:40:43 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-0: {default-scheduler } FailedScheduling: 0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:40:43 +0000 UTC - event for data-basic-h8jj7-us-east-1-rack-0-0: {k8s.io/minikube-hostpath_fv-az75-556_8a7df3ef-b96e-4204-b896-2402b7a93870 } ProvisioningSucceeded: Successfully provisioned volume pvc-e76fca9f-e226-41c8-bf81-9ed8b2067dce
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:40:43 +0000 UTC - event for data-basic-h8jj7-us-east-1-rack-0-0: {k8s.io/minikube-hostpath_fv-az75-556_8a7df3ef-b96e-4204-b896-2402b7a93870 } Provisioning: External provisioner is provisioning volume for claim "e2e-test-scyllacluster-mjgn4-6ndph/data-basic-h8jj7-us-east-1-rack-0-0"
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:40:43 +0000 UTC - event for data-basic-h8jj7-us-east-1-rack-0-0: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "k8s.io/minikube-hostpath" or manually created by system administrator
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:40:45 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-0: {default-scheduler } Scheduled: Successfully assigned e2e-test-scyllacluster-mjgn4-6ndph/basic-h8jj7-us-east-1-rack-0-0 to fv-az75-556
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:40:53 +0000 UTC - event for basic-h8jj7: {scyllacluster-controller } PodDisruptionBudgetCreated: PodDisruptionBudget e2e-test-scyllacluster-mjgn4-6ndph/basic-h8jj7 created
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:40:53 +0000 UTC - event for basic-h8jj7-client: {scyllacluster-controller } ServiceCreated: Service e2e-test-scyllacluster-mjgn4-6ndph/basic-h8jj7-client created
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:40:53 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-0: {scyllacluster-controller } ServiceCreated: Service e2e-test-scyllacluster-mjgn4-6ndph/basic-h8jj7-us-east-1-rack-0-0 created
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:40:53 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-1: {scyllacluster-controller } ServiceCreated: Service e2e-test-scyllacluster-mjgn4-6ndph/basic-h8jj7-us-east-1-rack-0-1 created
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:40:53 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-2: {scyllacluster-controller } ServiceCreated: Service e2e-test-scyllacluster-mjgn4-6ndph/basic-h8jj7-us-east-1-rack-0-2 created
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:41:09 +0000 UTC - event for nodeconfig-podinfo-d18759c5-d351-4c6a-9d5b-b0a6e9a7b5fb: {NodeConfigCM-controller } ConfigMapCreated: ConfigMap e2e-test-scyllacluster-mjgn4-6ndph/nodeconfig-podinfo-d18759c5-d351-4c6a-9d5b-b0a6e9a7b5fb created
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:41:10 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-0: {kubelet fv-az75-556} Created: Created container sidecar-injection
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:41:10 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-0: {kubelet fv-az75-556} Pulled: Container image "docker.io/scylladb/scylla-operator:ci" already present on machine
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:41:11 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-0: {kubelet fv-az75-556} Started: Started container sidecar-injection
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:41:19 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-0: {kubelet fv-az75-556} Pulled: Container image "docker.io/scylladb/scylla:4.6.2" already present on machine
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:41:20 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-0: {kubelet fv-az75-556} Created: Created container scylla
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:41:22 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-0: {kubelet fv-az75-556} Started: Started container scylla
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:41:22 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-0: {kubelet fv-az75-556} Pulled: Container image "docker.io/scylladb/scylla-manager-agent:2.6.3" already present on machine
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:41:22 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-0: {kubelet fv-az75-556} Created: Created container scylla-manager-agent
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:41:23 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-0: {kubelet fv-az75-556} Started: Started container scylla-manager-agent
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:41:25 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-0: {kubelet fv-az75-556} Unhealthy: Startup probe failed: Get "http://172.17.0.27:8080/healthz": dial tcp 172.17.0.27:8080: connect: connection refused
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:41:39 +0000 UTC - event for nodeconfig-podinfo-d18759c5-d351-4c6a-9d5b-b0a6e9a7b5fb: {NodeConfigCM-controller } ConfigMapUpdated: ConfigMap e2e-test-scyllacluster-mjgn4-6ndph/nodeconfig-podinfo-d18759c5-d351-4c6a-9d5b-b0a6e9a7b5fb updated
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:41:45 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-0: {kubelet fv-az75-556} Unhealthy: Startup probe failed: HTTP probe failed with statuscode: 503
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:42:05 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-0: {kubelet fv-az75-556} Unhealthy: Readiness probe failed: HTTP probe failed with statuscode: 500
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:42:25 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-0: {kubelet fv-az75-556} Unhealthy: Readiness probe failed: HTTP probe failed with statuscode: 503
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:43:10 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0: {statefulset-controller } SuccessfulCreate: create Pod basic-h8jj7-us-east-1-rack-0-1 in StatefulSet basic-h8jj7-us-east-1-rack-0 successful
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:43:10 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0: {statefulset-controller } SuccessfulCreate: create Claim data-basic-h8jj7-us-east-1-rack-0-1 Pod basic-h8jj7-us-east-1-rack-0-1 in StatefulSet basic-h8jj7-us-east-1-rack-0 success
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:43:10 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-1: {default-scheduler } FailedScheduling: 0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:43:10 +0000 UTC - event for data-basic-h8jj7-us-east-1-rack-0-1: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "k8s.io/minikube-hostpath" or manually created by system administrator
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:43:48 +0000 UTC - event for data-basic-h8jj7-us-east-1-rack-0-1: {k8s.io/minikube-hostpath_fv-az75-556_7fdc1012-cb5f-46b9-96d3-10d9d324748d } Provisioning: External provisioner is provisioning volume for claim "e2e-test-scyllacluster-mjgn4-6ndph/data-basic-h8jj7-us-east-1-rack-0-1"
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:43:49 +0000 UTC - event for data-basic-h8jj7-us-east-1-rack-0-1: {k8s.io/minikube-hostpath_fv-az75-556_7fdc1012-cb5f-46b9-96d3-10d9d324748d } ProvisioningSucceeded: Successfully provisioned volume pvc-f7c1b3f2-b040-46ab-bd8f-35399d69e8ef
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:43:56 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-1: {default-scheduler } Scheduled: Successfully assigned e2e-test-scyllacluster-mjgn4-6ndph/basic-h8jj7-us-east-1-rack-0-1 to fv-az75-556
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:43:56 +0000 UTC - event for nodeconfig-podinfo-84[660](https://github.com/scylladb/scylla-operator/actions/runs/3066479399/jobs/4951738619#step:12:661)45e-6a70-4051-ac54-f8086dcf25f3: {NodeConfigCM-controller } ConfigMapCreated: ConfigMap e2e-test-scyllacluster-mjgn4-6ndph/nodeconfig-podinfo-8466045e-6a70-4051-ac54-f8086dcf25f3 created
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:44:03 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-1: {kubelet fv-az75-556} Pulled: Container image "docker.io/scylladb/scylla-operator:ci" already present on machine
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:44:03 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-1: {kubelet fv-az75-556} Created: Created container sidecar-injection
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:44:04 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-1: {kubelet fv-az75-556} Started: Started container sidecar-injection
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:44:13 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-1: {kubelet fv-az75-556} Pulled: Container image "docker.io/scylladb/scylla-manager-agent:2.6.3" already present on machine
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:44:13 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-1: {kubelet fv-az75-556} Pulled: Container image "docker.io/scylladb/scylla:4.6.2" already present on machine
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:44:13 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-1: {kubelet fv-az75-556} Created: Created container scylla
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:44:13 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-1: {kubelet fv-az75-556} Started: Started container scylla
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:44:14 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-1: {kubelet fv-az75-556} Created: Created container scylla-manager-agent
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:44:14 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-1: {kubelet fv-az75-556} Started: Started container scylla-manager-agent
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:44:16 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-1: {kubelet fv-az75-556} Unhealthy: Startup probe failed: Get "http://172.17.0.29:8080/healthz": dial tcp 172.17.0.29:8080: connect: connection refused
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:44:19 +0000 UTC - event for nodeconfig-podinfo-8466045e-6a70-4051-ac54-f8086dcf25f3: {NodeConfigCM-controller } ConfigMapUpdated: ConfigMap e2e-test-scyllacluster-mjgn4-6ndph/nodeconfig-podinfo-8466045e-6a70-4051-ac54-f8086dcf25f3 updated
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:44:26 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-1: {kubelet fv-az75-556} Unhealthy: Startup probe failed: HTTP probe failed with statuscode: 503
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:44:36 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-1: {kubelet fv-az75-556} Unhealthy: Readiness probe failed: HTTP probe failed with statuscode: 500
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:44:46 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-1: {kubelet fv-az75-556} Unhealthy: Readiness probe failed: HTTP probe failed with statuscode: 503
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:46:26 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0: {statefulset-controller } SuccessfulCreate: create Claim data-basic-h8jj7-us-east-1-rack-0-2 Pod basic-h8jj7-us-east-1-rack-0-2 in StatefulSet basic-h8jj7-us-east-1-rack-0 success
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:46:26 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0: {statefulset-controller } SuccessfulCreate: create Pod basic-h8jj7-us-east-1-rack-0-2 in StatefulSet basic-h8jj7-us-east-1-rack-0 successful
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:46:26 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-2: {default-scheduler } FailedScheduling: 0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:46:26 +0000 UTC - event for data-basic-h8jj7-us-east-1-rack-0-2: {k8s.io/minikube-hostpath_fv-az75-556_7fdc1012-cb5f-46b9-96d3-10d9d324748d } ProvisioningSucceeded: Successfully provisioned volume pvc-a2509279-0da9-4bc6-9974-77953db256ab
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:46:26 +0000 UTC - event for data-basic-h8jj7-us-east-1-rack-0-2: {k8s.io/minikube-hostpath_fv-az75-556_7fdc1012-cb5f-46b9-96d3-10d9d324748d } Provisioning: External provisioner is provisioning volume for claim "e2e-test-scyllacluster-mjgn4-6ndph/data-basic-h8jj7-us-east-1-rack-0-2"
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:46:26 +0000 UTC - event for data-basic-h8jj7-us-east-1-rack-0-2: {persistentvolume-controller } ExternalProvisioning: waiting for a volume to be created, either by external provisioner "k8s.io/minikube-hostpath" or manually created by system administrator
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:46:28 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-2: {default-scheduler } Scheduled: Successfully assigned e2e-test-scyllacluster-mjgn4-6ndph/basic-h8jj7-us-east-1-rack-0-2 to fv-az75-556
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:46:29 +0000 UTC - event for nodeconfig-podinfo-a409a6bc-64b2-4e5f-98d6-071c9c82a87e: {NodeConfigCM-controller } ConfigMapCreated: ConfigMap e2e-test-scyllacluster-mjgn4-6ndph/nodeconfig-podinfo-a409a6bc-64b2-4e5f-98d6-071c9c82a87e created
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:46:30 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-2: {kubelet fv-az75-556} Pulled: Container image "docker.io/scylladb/scylla-operator:ci" already present on machine
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:46:30 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-2: {kubelet fv-az75-556} Created: Created container sidecar-injection
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:46:31 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-2: {kubelet fv-az75-556} Started: Started container sidecar-injection
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:46:40 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-2: {kubelet fv-az75-556} Pulled: Container image "docker.io/scylladb/scylla:4.6.2" already present on machine
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:46:40 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-2: {kubelet fv-az75-556} Created: Created container scylla
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:46:41 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-2: {kubelet fv-az75-556} Started: Started container scylla
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:46:41 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-2: {kubelet fv-az75-556} Pulled: Container image "docker.io/scylladb/scylla-manager-agent:2.6.3" already present on machine
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:46:42 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-2: {kubelet fv-az75-556} Created: Created container scylla-manager-agent
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:46:42 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-2: {kubelet fv-az75-556} Started: Started container scylla-manager-agent
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:46:43 +0000 UTC - event for nodeconfig-podinfo-a409a6bc-64b2-4e5f-98d6-071c9c82a87e: {NodeConfigCM-controller } ConfigMapUpdated: ConfigMap e2e-test-scyllacluster-mjgn4-6ndph/nodeconfig-podinfo-a409a6bc-64b2-4e5f-98d6-071c9c82a87e updated
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:46:48 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-2: {kubelet fv-az75-556} Unhealthy: Startup probe failed: HTTP probe failed with statuscode: 503
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:46:58 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-2: {kubelet fv-az75-556} Unhealthy: Readiness probe failed: HTTP probe failed with statuscode: 503
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:48:55 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0: {statefulset-controller } SuccessfulDelete: delete Pod basic-h8jj7-us-east-1-rack-0-2 in StatefulSet basic-h8jj7-us-east-1-rack-0 successful
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:48:55 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0: {scyllacluster-controller } StatefulSetUpdated: StatefulSet e2e-test-scyllacluster-mjgn4-6ndph/basic-h8jj7-us-east-1-rack-0 updated
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:48:55 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-2: {kubelet fv-az75-556} Killing: Stopping container scylla-manager-agent
    Sep 16 08:49:00.875: INFO: At 2022-09-16 08:48:55 +0000 UTC - event for basic-h8jj7-us-east-1-rack-0-2: {kubelet fv-az75-556} Killing: Stopping container scylla
    STEP: Collecting dumps from namespace "e2e-test-scyllacluster-mjgn4-6ndph". 09/16/22 08:49:00.875
    STEP: Destroying namespace "e2e-test-scyllacluster-mjgn4-6ndph". 09/16/22 08:49:01.473
    STEP: Waiting for namespace "e2e-test-scyllacluster-mjgn4-6ndph" to be removed. 09/16/22 08:49:01.479
  << End Captured GinkgoWriter Output

tnozicka avatar Sep 16 '22 10:09 tnozicka

flaked on master as well https://github.com/scylladb/scylla-operator/actions/runs/3064280032/jobs/4947265598#step:12:954

tnozicka avatar Sep 16 '22 10:09 tnozicka

flaking a lot, next one https://github.com/scylladb/scylla-operator/actions/runs/3067025626/jobs/4952910620#step:12:700

tnozicka avatar Sep 16 '22 12:09 tnozicka

I also saw a lot of these on the TLS PR

tnozicka avatar Sep 16 '22 12:09 tnozicka

flaking a lot, last one in https://github.com/scylladb/scylla-operator/actions/runs/3086220438/jobs/4990391763#step:12:261 https://github.com/scylladb/scylla-operator/suites/8367743162/artifacts/369018669

tnozicka avatar Sep 20 '22 05:09 tnozicka

fyi, major upgrades also flake, haven't looked whether that's related

tnozicka avatar Sep 22 '22 09:09 tnozicka

I found three causes of the flakiness so far, all of them might be shared across different tests:

  1. #1045
  2. Hitting watch cache in initial list operation in https://github.com/scylladb/scylla-operator/blob/d84a8b1160804a1daf8a28f8ce977c393398f778/test/e2e/utils/helpers.go#L172. @tnozicka is meant to send a hotfix.
  3. #1047

rzetelskik avatar Sep 29 '22 14:09 rzetelskik

STEP: Waiting for the ScyllaCluster to re-deploy 09/16/22 08:48:55.746 Sep 16 08:48:55.748: INFO: ScyllaCluster e2e-test-scyllacluster-mjgn4-6ndph/basic-h8jj7 (RV=7123) is rolled out

Fixed by https://github.com/scylladb/scylla-operator/pull/1049

tnozicka avatar Oct 19 '22 11:10 tnozicka