che
che copied to clipboard
eclipse-che
0/4 nodes are available: 4 pod has unbound immediate PersistentVolumeClaims
Summary
my chectl version chectl/0.0.20220801-next.24265f6 darwin-x64 node-v16.13.2
i installed eclipse-che using below command
chectl server:deploy --che-operator-cr-patch-yaml=/Users/divine/Documents/office_tasks/TAP-4540/terraformCheJune17/eclipseche_yaml/che-operator-cr-patch.yaml --platform=k8s --installer=operator --debug --k8spoderrorrechecktimeout=1500000 --domain=eclipseche-dchelladurai-chejune15.calatrava.vmware.com --k8spodreadytimeout=1500000
i just logged-in into eclipse-che dashboard and cloned 1 workspace.
below are my pv,pvc,storageclass
dchelladura-a01:vtapbotservice divine$ kubectl get pv,pvc,storageclass -n divine-chelladurai-che-pmjg9u
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
persistentvolume/pvc-80278a68-7f9a-4e24-83b5-ff4fdf993a35 10Gi RWO Delete Bound divine-chelladurai-che-pmjg9u/claim-devworkspace vmc-workload-storage-policy-cluster-1 6m4s
persistentvolume/pvc-d5892021-9ea0-4125-8cce-5ed82bcf7899 10Gi RWO Delete Bound d-sindhuja-che-a6q94i/claim-devworkspace vmc-workload-storage-policy-cluster-1 3h22m
persistentvolume/pvc-e01098d1-e183-4d44-ba9c-d1086de34636 3Gi RWO Delete Bound default/task-pv-claim vmc-workload-storage-policy-cluster-1 17d
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/claim-devworkspace Bound pvc-80278a68-7f9a-4e24-83b5-ff4fdf993a35 10Gi RWO vmc-workload-storage-policy-cluster-1 6m6s
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
storageclass.storage.k8s.io/vmc-workload-storage-policy-cluster-1 (default) csi.vsphere.vmware.com Delete Immediate true 51d
dchelladura-a01:vtapbotservice divine$
i'm frequently receive this error. i had to wait for couple of hours for this to goaway. i dont know what is the reason for this error
dchelladura-a01:vtapbotservice divine$ kubectl get events -w -n divine-chelladurai-che-pmjg9u
LAST SEEN TYPE REASON OBJECT MESSAGE
91s Normal ExternalProvisioning persistentvolumeclaim/claim-devworkspace waiting for a volume to be created, either by external provisioner "csi.vsphere.vmware.com" or manually created by system administrator
91s Normal Provisioning persistentvolumeclaim/claim-devworkspace External provisioner is provisioning volume for claim "divine-chelladurai-che-pmjg9u/claim-devworkspace"
90s Normal ProvisioningSucceeded persistentvolumeclaim/claim-devworkspace Successfully provisioned volume pvc-80278a68-7f9a-4e24-83b5-ff4fdf993a35
29s Normal Scheduled pod/workspace7722130dbd9049ee-57c6b44b8d-drw79 Successfully assigned divine-chelladurai-che-pmjg9u/workspace7722130dbd9049ee-57c6b44b8d-drw79 to tkc-workers-wnnlk-b55cc6bdd-d6767
28s Normal SuccessfulAttachVolume pod/workspace7722130dbd9049ee-57c6b44b8d-drw79 AttachVolume.Attach succeeded for volume "pvc-80278a68-7f9a-4e24-83b5-ff4fdf993a35"
20s Normal Pulling pod/workspace7722130dbd9049ee-57c6b44b8d-drw79 Pulling image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next"
19s Normal Pulled pod/workspace7722130dbd9049ee-57c6b44b8d-drw79 Successfully pulled image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next" in 496.608681ms
19s Normal Created pod/workspace7722130dbd9049ee-57c6b44b8d-drw79 Created container remote-runtime-injector
19s Normal Started pod/workspace7722130dbd9049ee-57c6b44b8d-drw79 Started container remote-runtime-injector
18s Normal Pulling pod/workspace7722130dbd9049ee-57c6b44b8d-drw79 Pulling image "quay.io/devfile/project-clone:v0.15.2"
18s Normal Pulled pod/workspace7722130dbd9049ee-57c6b44b8d-drw79 Successfully pulled image "quay.io/devfile/project-clone:v0.15.2" in 346.570157ms
18s Normal Created pod/workspace7722130dbd9049ee-57c6b44b8d-drw79 Created container project-clone
18s Normal Started pod/workspace7722130dbd9049ee-57c6b44b8d-drw79 Started container project-clone
29s Normal SuccessfulCreate replicaset/workspace7722130dbd9049ee-57c6b44b8d Created pod: workspace7722130dbd9049ee-57c6b44b8d-drw79
14s Normal SuccessfulDelete replicaset/workspace7722130dbd9049ee-57c6b44b8d Deleted pod: workspace7722130dbd9049ee-57c6b44b8d-drw79
91s Warning FailedScheduling pod/workspace7722130dbd9049ee-5f5fbbc678-khd9s 0/4 nodes are available: 4 pod has unbound immediate PersistentVolumeClaims.
91s Normal SuccessfulCreate replicaset/workspace7722130dbd9049ee-5f5fbbc678 Created pod: workspace7722130dbd9049ee-5f5fbbc678-khd9s
90s Normal SuccessfulDelete replicaset/workspace7722130dbd9049ee-5f5fbbc678 Deleted pod: workspace7722130dbd9049ee-5f5fbbc678-khd9s
91s Normal ScalingReplicaSet deployment/workspace7722130dbd9049ee Scaled up replica set workspace7722130dbd9049ee-5f5fbbc678 to 1
90s Normal ScalingReplicaSet deployment/workspace7722130dbd9049ee Scaled down replica set workspace7722130dbd9049ee-5f5fbbc678 to 0
29s Normal ScalingReplicaSet deployment/workspace7722130dbd9049ee Scaled up replica set workspace7722130dbd9049ee-57c6b44b8d to 1
14s Normal ScalingReplicaSet deployment/workspace7722130dbd9049ee Scaled down replica set workspace7722130dbd9049ee-57c6b44b8d to 0
0s Normal ScalingReplicaSet deployment/workspace7722130dbd9049ee Scaled up replica set workspace7722130dbd9049ee-5f5fbbc678 to 1
0s Normal Scheduled pod/workspace7722130dbd9049ee-5f5fbbc678-n7kh7 Successfully assigned divine-chelladurai-che-pmjg9u/workspace7722130dbd9049ee-5f5fbbc678-n7kh7 to tkc-workers-wnnlk-b55cc6bdd-d6767
0s Normal SuccessfulCreate replicaset/workspace7722130dbd9049ee-5f5fbbc678 Created pod: workspace7722130dbd9049ee-5f5fbbc678-n7kh7
0s Normal SuccessfulAttachVolume pod/workspace7722130dbd9049ee-5f5fbbc678-n7kh7 AttachVolume.Attach succeeded for volume "pvc-80278a68-7f9a-4e24-83b5-ff4fdf993a35"
0s Normal Pulling pod/workspace7722130dbd9049ee-5f5fbbc678-n7kh7 Pulling image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next"
0s Normal Pulled pod/workspace7722130dbd9049ee-5f5fbbc678-n7kh7 Successfully pulled image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next" in 330.553683ms
0s Normal Created pod/workspace7722130dbd9049ee-5f5fbbc678-n7kh7 Created container remote-runtime-injector
0s Normal Started pod/workspace7722130dbd9049ee-5f5fbbc678-n7kh7 Started container remote-runtime-injector
0s Normal Pulling pod/workspace7722130dbd9049ee-5f5fbbc678-n7kh7 Pulling image "quay.io/devfile/project-clone:v0.15.2"
0s Normal Pulled pod/workspace7722130dbd9049ee-5f5fbbc678-n7kh7 Successfully pulled image "quay.io/devfile/project-clone:v0.15.2" in 345.867809ms
0s Normal Created pod/workspace7722130dbd9049ee-5f5fbbc678-n7kh7 Created container project-clone
0s Normal Started pod/workspace7722130dbd9049ee-5f5fbbc678-n7kh7 Started container project-clone
0s Normal ScalingReplicaSet deployment/workspace7722130dbd9049ee Scaled down replica set workspace7722130dbd9049ee-5f5fbbc678 to 0
0s Normal SuccessfulDelete replicaset/workspace7722130dbd9049ee-5f5fbbc678 Deleted pod: workspace7722130dbd9049ee-5f5fbbc678-n7kh7
0s Normal ScalingReplicaSet deployment/workspace7722130dbd9049ee Scaled up replica set workspace7722130dbd9049ee-5f5fbbc678 to 1
0s Normal SuccessfulCreate replicaset/workspace7722130dbd9049ee-5f5fbbc678 Created pod: workspace7722130dbd9049ee-5f5fbbc678-4jrsb
0s Normal Scheduled pod/workspace7722130dbd9049ee-5f5fbbc678-4jrsb Successfully assigned divine-chelladurai-che-pmjg9u/workspace7722130dbd9049ee-5f5fbbc678-4jrsb to tkc-workers-wnnlk-b55cc6bdd-d6767
0s Normal ScalingReplicaSet deployment/workspace7722130dbd9049ee Scaled down replica set workspace7722130dbd9049ee-5f5fbbc678 to 0
0s Normal SuccessfulDelete replicaset/workspace7722130dbd9049ee-5f5fbbc678 Deleted pod: workspace7722130dbd9049ee-5f5fbbc678-4jrsb
0s Normal ScalingReplicaSet deployment/workspace7722130dbd9049ee Scaled up replica set workspace7722130dbd9049ee-5f5fbbc678 to 1
0s Normal SuccessfulCreate replicaset/workspace7722130dbd9049ee-5f5fbbc678 Created pod: workspace7722130dbd9049ee-5f5fbbc678-t92gv
0s Normal Scheduled pod/workspace7722130dbd9049ee-5f5fbbc678-t92gv Successfully assigned divine-chelladurai-che-pmjg9u/workspace7722130dbd9049ee-5f5fbbc678-t92gv to tkc-workers-wnnlk-b55cc6bdd-7k767
0s Warning FailedAttachVolume pod/workspace7722130dbd9049ee-5f5fbbc678-t92gv Multi-Attach error for volume "pvc-80278a68-7f9a-4e24-83b5-ff4fdf993a35" Volume is already exclusively attached to one node and can't be attached to another
0s Normal Pulling pod/workspace7722130dbd9049ee-5f5fbbc678-4jrsb Pulling image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next"
0s Normal Pulled pod/workspace7722130dbd9049ee-5f5fbbc678-4jrsb Successfully pulled image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next" in 472.644726ms
0s Normal Created pod/workspace7722130dbd9049ee-5f5fbbc678-4jrsb Created container remote-runtime-injector
0s Warning Failed pod/workspace7722130dbd9049ee-5f5fbbc678-4jrsb Error: sandbox container "eb89576a82303e752f180c5f5d394d3a3833da65036b5c36e288c0440c724c97" is not running
0s Normal SuccessfulAttachVolume pod/workspace7722130dbd9049ee-5f5fbbc678-t92gv AttachVolume.Attach succeeded for volume "pvc-80278a68-7f9a-4e24-83b5-ff4fdf993a35"
0s Normal Pulling pod/workspace7722130dbd9049ee-5f5fbbc678-t92gv Pulling image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next"
0s Normal Pulled pod/workspace7722130dbd9049ee-5f5fbbc678-t92gv Successfully pulled image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next" in 340.879788ms
0s Normal Created pod/workspace7722130dbd9049ee-5f5fbbc678-t92gv Created container remote-runtime-injector
0s Normal Started pod/workspace7722130dbd9049ee-5f5fbbc678-t92gv Started container remote-runtime-injector
0s Normal Pulling pod/workspace7722130dbd9049ee-5f5fbbc678-t92gv Pulling image "quay.io/devfile/project-clone:v0.15.2"
0s Normal Pulled pod/workspace7722130dbd9049ee-5f5fbbc678-t92gv Successfully pulled image "quay.io/devfile/project-clone:v0.15.2" in 674.508557ms
0s Normal Created pod/workspace7722130dbd9049ee-5f5fbbc678-t92gv Created container project-clone
0s Normal Started pod/workspace7722130dbd9049ee-5f5fbbc678-t92gv Started container project-clone
0s Normal ScalingReplicaSet deployment/workspace7722130dbd9049ee Scaled down replica set workspace7722130dbd9049ee-5f5fbbc678 to 0
0s Normal SuccessfulDelete replicaset/workspace7722130dbd9049ee-5f5fbbc678 Deleted pod: workspace7722130dbd9049ee-5f5fbbc678-t92gv
0s Normal ScalingReplicaSet deployment/workspace7722130dbd9049ee Scaled up replica set workspace7722130dbd9049ee-5f5fbbc678 to 1
0s Normal SuccessfulCreate replicaset/workspace7722130dbd9049ee-5f5fbbc678 Created pod: workspace7722130dbd9049ee-5f5fbbc678-gbrz5
0s Normal Scheduled pod/workspace7722130dbd9049ee-5f5fbbc678-gbrz5 Successfully assigned divine-chelladurai-che-pmjg9u/workspace7722130dbd9049ee-5f5fbbc678-gbrz5 to tkc-workers-wnnlk-b55cc6bdd-d6767
0s Warning FailedAttachVolume pod/workspace7722130dbd9049ee-5f5fbbc678-gbrz5 Multi-Attach error for volume "pvc-80278a68-7f9a-4e24-83b5-ff4fdf993a35" Volume is already exclusively attached to one node and can't be attached to another
0s Normal SuccessfulAttachVolume pod/workspace7722130dbd9049ee-5f5fbbc678-gbrz5 AttachVolume.Attach succeeded for volume "pvc-80278a68-7f9a-4e24-83b5-ff4fdf993a35"
0s Normal Pulling pod/workspace7722130dbd9049ee-5f5fbbc678-gbrz5 Pulling image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next"
0s Normal Pulled pod/workspace7722130dbd9049ee-5f5fbbc678-gbrz5 Successfully pulled image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next" in 454.743658ms
0s Normal Created pod/workspace7722130dbd9049ee-5f5fbbc678-gbrz5 Created container remote-runtime-injector
0s Normal Started pod/workspace7722130dbd9049ee-5f5fbbc678-gbrz5 Started container remote-runtime-injector
0s Normal Pulling pod/workspace7722130dbd9049ee-5f5fbbc678-gbrz5 Pulling image "quay.io/devfile/project-clone:v0.15.2"
0s Normal Pulled pod/workspace7722130dbd9049ee-5f5fbbc678-gbrz5 Successfully pulled image "quay.io/devfile/project-clone:v0.15.2" in 344.960292ms
0s Normal Created pod/workspace7722130dbd9049ee-5f5fbbc678-gbrz5 Created container project-clone
0s Normal Started pod/workspace7722130dbd9049ee-5f5fbbc678-gbrz5 Started container project-clone
0s Normal ScalingReplicaSet deployment/workspace7722130dbd9049ee Scaled down replica set workspace7722130dbd9049ee-5f5fbbc678 to 0
0s Normal SuccessfulDelete replicaset/workspace7722130dbd9049ee-5f5fbbc678 Deleted pod: workspace7722130dbd9049ee-5f5fbbc678-gbrz5
0s Normal ScalingReplicaSet deployment/workspace7722130dbd9049ee Scaled up replica set workspace7722130dbd9049ee-5f5fbbc678 to 1
0s Normal SuccessfulCreate replicaset/workspace7722130dbd9049ee-5f5fbbc678 Created pod: workspace7722130dbd9049ee-5f5fbbc678-zsss9
0s Normal Scheduled pod/workspace7722130dbd9049ee-5f5fbbc678-zsss9 Successfully assigned divine-chelladurai-che-pmjg9u/workspace7722130dbd9049ee-5f5fbbc678-zsss9 to tkc-workers-wnnlk-b55cc6bdd-7k767
0s Warning FailedAttachVolume pod/workspace7722130dbd9049ee-5f5fbbc678-zsss9 Multi-Attach error for volume "pvc-80278a68-7f9a-4e24-83b5-ff4fdf993a35" Volume is already exclusively attached to one node and can't be attached to another
0s Normal ScalingReplicaSet deployment/workspace7722130dbd9049ee Scaled down replica set workspace7722130dbd9049ee-5f5fbbc678 to 0
0s Normal SuccessfulDelete replicaset/workspace7722130dbd9049ee-5f5fbbc678 Deleted pod: workspace7722130dbd9049ee-5f5fbbc678-zsss9
0s Normal ScalingReplicaSet deployment/workspace7722130dbd9049ee Scaled up replica set workspace7722130dbd9049ee-5f5fbbc678 to 1
0s Normal SuccessfulCreate replicaset/workspace7722130dbd9049ee-5f5fbbc678 Created pod: workspace7722130dbd9049ee-5f5fbbc678-ztjlb
0s Normal Scheduled pod/workspace7722130dbd9049ee-5f5fbbc678-ztjlb Successfully assigned divine-chelladurai-che-pmjg9u/workspace7722130dbd9049ee-5f5fbbc678-ztjlb to tkc-workers-wnnlk-b55cc6bdd-h75s7
0s Warning FailedAttachVolume pod/workspace7722130dbd9049ee-5f5fbbc678-ztjlb Multi-Attach error for volume "pvc-80278a68-7f9a-4e24-83b5-ff4fdf993a35" Volume is already exclusively attached to one node and can't be attached to another
0s Normal ScalingReplicaSet deployment/workspace7722130dbd9049ee Scaled down replica set workspace7722130dbd9049ee-5f5fbbc678 to 0
0s Normal SuccessfulDelete replicaset/workspace7722130dbd9049ee-5f5fbbc678 Deleted pod: workspace7722130dbd9049ee-5f5fbbc678-ztjlb
0s Normal ScalingReplicaSet deployment/workspace7722130dbd9049ee Scaled up replica set workspace7722130dbd9049ee-5f5fbbc678 to 1
0s Normal SuccessfulCreate replicaset/workspace7722130dbd9049ee-5f5fbbc678 Created pod: workspace7722130dbd9049ee-5f5fbbc678-94nwf
0s Normal Scheduled pod/workspace7722130dbd9049ee-5f5fbbc678-94nwf Successfully assigned divine-chelladurai-che-pmjg9u/workspace7722130dbd9049ee-5f5fbbc678-94nwf to tkc-workers-wnnlk-b55cc6bdd-d6767
0s Normal SuccessfulAttachVolume pod/workspace7722130dbd9049ee-5f5fbbc678-ztjlb AttachVolume.Attach succeeded for volume "pvc-80278a68-7f9a-4e24-83b5-ff4fdf993a35"
0s Warning FailedAttachVolume pod/workspace7722130dbd9049ee-5f5fbbc678-94nwf Multi-Attach error for volume "pvc-80278a68-7f9a-4e24-83b5-ff4fdf993a35" Volume is already exclusively attached to one node and can't be attached to another
0s Normal ScalingReplicaSet deployment/workspace7722130dbd9049ee Scaled down replica set workspace7722130dbd9049ee-5f5fbbc678 to 0
0s Normal SuccessfulDelete replicaset/workspace7722130dbd9049ee-5f5fbbc678 Deleted pod: workspace7722130dbd9049ee-5f5fbbc678-94nwf
0s Normal ScalingReplicaSet deployment/workspace7722130dbd9049ee Scaled up replica set workspace7722130dbd9049ee-5f5fbbc678 to 1
0s Normal SuccessfulCreate replicaset/workspace7722130dbd9049ee-5f5fbbc678 Created pod: workspace7722130dbd9049ee-5f5fbbc678-lszjj
0s Normal Scheduled pod/workspace7722130dbd9049ee-5f5fbbc678-lszjj Successfully assigned divine-chelladurai-che-pmjg9u/workspace7722130dbd9049ee-5f5fbbc678-lszjj to tkc-workers-wnnlk-b55cc6bdd-h75s7
0s Normal ScalingReplicaSet deployment/workspace7722130dbd9049ee Scaled down replica set workspace7722130dbd9049ee-5f5fbbc678 to 0
0s Normal SuccessfulDelete replicaset/workspace7722130dbd9049ee-5f5fbbc678 Deleted pod: workspace7722130dbd9049ee-5f5fbbc678-lszjj
0s Normal ScalingReplicaSet deployment/workspace7722130dbd9049ee Scaled up replica set workspace7722130dbd9049ee-5f5fbbc678 to 1
0s Normal SuccessfulCreate replicaset/workspace7722130dbd9049ee-5f5fbbc678 (combined from similar events): Created pod: workspace7722130dbd9049ee-5f5fbbc678-t5k5j
0s Normal Scheduled pod/workspace7722130dbd9049ee-5f5fbbc678-t5k5j Successfully assigned divine-chelladurai-che-pmjg9u/workspace7722130dbd9049ee-5f5fbbc678-t5k5j to tkc-workers-wnnlk-b55cc6bdd-d6767
0s Normal ScalingReplicaSet deployment/workspace7722130dbd9049ee Scaled down replica set workspace7722130dbd9049ee-5f5fbbc678 to 0
0s Normal SuccessfulDelete replicaset/workspace7722130dbd9049ee-5f5fbbc678 (combined from similar events): Deleted pod: workspace7722130dbd9049ee-5f5fbbc678-t5k5j
0s Normal ScalingReplicaSet deployment/workspace7722130dbd9049ee Scaled up replica set workspace7722130dbd9049ee-5f5fbbc678 to 1
0s Normal SuccessfulCreate replicaset/workspace7722130dbd9049ee-5f5fbbc678 (combined from similar events): Created pod: workspace7722130dbd9049ee-5f5fbbc678-brt29
0s Normal Scheduled pod/workspace7722130dbd9049ee-5f5fbbc678-brt29 Successfully assigned divine-chelladurai-che-pmjg9u/workspace7722130dbd9049ee-5f5fbbc678-brt29 to tkc-workers-wnnlk-b55cc6bdd-d6767
0s Normal ScalingReplicaSet deployment/workspace7722130dbd9049ee Scaled down replica set workspace7722130dbd9049ee-5f5fbbc678 to 0
0s Normal SuccessfulDelete replicaset/workspace7722130dbd9049ee-5f5fbbc678 (combined from similar events): Deleted pod: workspace7722130dbd9049ee-5f5fbbc678-brt29
0s Normal ScalingReplicaSet deployment/workspace7722130dbd9049ee Scaled up replica set workspace7722130dbd9049ee-5f5fbbc678 to 1
0s Normal SuccessfulCreate replicaset/workspace7722130dbd9049ee-5f5fbbc678 (combined from similar events): Created pod: workspace7722130dbd9049ee-5f5fbbc678-dzz9r
0s Normal Scheduled pod/workspace7722130dbd9049ee-5f5fbbc678-dzz9r Successfully assigned divine-chelladurai-che-pmjg9u/workspace7722130dbd9049ee-5f5fbbc678-dzz9r to tkc-workers-wnnlk-b55cc6bdd-7k767
0s Normal ScalingReplicaSet deployment/workspace7722130dbd9049ee Scaled down replica set workspace7722130dbd9049ee-5f5fbbc678 to 0
0s Normal SuccessfulDelete replicaset/workspace7722130dbd9049ee-5f5fbbc678 (combined from similar events): Deleted pod: workspace7722130dbd9049ee-5f5fbbc678-dzz9r
0s Normal SuccessfulCreate replicaset/workspace7722130dbd9049ee-5f5fbbc678 (combined from similar events): Created pod: workspace7722130dbd9049ee-5f5fbbc678-pj97z
0s Normal Scheduled pod/workspace7722130dbd9049ee-5f5fbbc678-pj97z Successfully assigned divine-chelladurai-che-pmjg9u/workspace7722130dbd9049ee-5f5fbbc678-pj97z to tkc-workers-wnnlk-b55cc6bdd-d6767
0s Warning FailedMount pod/workspace7722130dbd9049ee-5f5fbbc678-pj97z MountVolume.SetUp failed for volume "automount-secret-che-server-cert" : failed to sync secret cache: timed out waiting for the condition
0s Warning FailedMount pod/workspace7722130dbd9049ee-5f5fbbc678-pj97z MountVolume.SetUp failed for volume "automount-configmap-che-trusted-ca-certs" : failed to sync configmap cache: timed out waiting for the condition
0s Warning FailedMount pod/workspace7722130dbd9049ee-5f5fbbc678-pj97z MountVolume.SetUp failed for volume "che-gateway" : failed to sync configmap cache: timed out waiting for the condition
0s Warning FailedMount pod/workspace7722130dbd9049ee-5f5fbbc678-zsss9 Unable to attach or mount volumes: unmounted volumes=[claim-devworkspace automount-configmap-devworkspace-gitconfig automount-secret-che-server-cert remote-endpoint automount-secret-devworkspace-merged-git-credentials workspace-metadata automount-secret-user-profile che-gateway automount-configmap-che-trusted-ca-certs kube-api-access-jjqsr], unattached volumes=[claim-devworkspace automount-configmap-devworkspace-gitconfig automount-secret-che-server-cert remote-endpoint automount-secret-devworkspace-merged-git-credentials workspace-metadata automount-secret-user-profile che-gateway automount-configmap-che-trusted-ca-certs kube-api-access-jjqsr]: timed out waiting for the condition
0s Warning FailedMount pod/workspace7722130dbd9049ee-5f5fbbc678-ztjlb Unable to attach or mount volumes: unmounted volumes=[kube-api-access-zr8xs automount-secret-user-profile workspace-metadata claim-devworkspace remote-endpoint automount-configmap-devworkspace-gitconfig automount-configmap-che-trusted-ca-certs che-gateway automount-secret-devworkspace-merged-git-credentials automount-secret-che-server-cert], unattached volumes=[kube-api-access-zr8xs automount-secret-user-profile workspace-metadata claim-devworkspace remote-endpoint automount-configmap-devworkspace-gitconfig automount-configmap-che-trusted-ca-certs che-gateway automount-secret-devworkspace-merged-git-credentials automount-secret-che-server-cert]: timed out waiting for the condition
0s Normal SuccessfulDelete replicaset/workspace7722130dbd9049ee-5f5fbbc678 (combined from similar events): Deleted pod: workspace7722130dbd9049ee-5f5fbbc678-pj97z
0s Normal Scheduled pod/workspace7722130dbd9049ee-5f5fbbc678-s9r62 Successfully assigned divine-chelladurai-che-pmjg9u/workspace7722130dbd9049ee-5f5fbbc678-s9r62 to tkc-workers-wnnlk-b55cc6bdd-d6767
0s Warning FailedMount pod/workspace7722130dbd9049ee-5f5fbbc678-94nwf Unable to attach or mount volumes: unmounted volumes=[workspace-metadata automount-secret-devworkspace-merged-git-credentials automount-configmap-devworkspace-gitconfig automount-secret-user-profile kube-api-access-qdc79 remote-endpoint automount-configmap-che-trusted-ca-certs che-gateway automount-secret-che-server-cert claim-devworkspace], unattached volumes=[workspace-metadata automount-secret-devworkspace-merged-git-credentials automount-configmap-devworkspace-gitconfig automount-secret-user-profile kube-api-access-qdc79 remote-endpoint automount-configmap-che-trusted-ca-certs che-gateway automount-secret-che-server-cert claim-devworkspace]: timed out waiting for the condition
0s Warning FailedAttachVolume pod/workspace7722130dbd9049ee-5f5fbbc678-94nwf AttachVolume.Attach failed for volume "pvc-80278a68-7f9a-4e24-83b5-ff4fdf993a35" : Attach timeout for volume df63f994-eaeb-47a8-ac23-0e222cac3b10-80278a68-7f9a-4e24-83b5-ff4fdf993a35
0s Warning FailedMount pod/workspace7722130dbd9049ee-5f5fbbc678-lszjj Unable to attach or mount volumes: unmounted volumes=[automount-secret-user-profile kube-api-access-m5xcx claim-devworkspace automount-configmap-che-trusted-ca-certs automount-secret-che-server-cert automount-configmap-devworkspace-gitconfig remote-endpoint workspace-metadata che-gateway automount-secret-devworkspace-merged-git-credentials], unattached volumes=[automount-secret-user-profile kube-api-access-m5xcx claim-devworkspace automount-configmap-che-trusted-ca-certs automount-secret-che-server-cert automount-configmap-devworkspace-gitconfig remote-endpoint workspace-metadata che-gateway automount-secret-devworkspace-merged-git-credentials]: timed out waiting for the condition
0s Normal SuccessfulCreate replicaset/workspace7722130dbd9049ee-57c6b44b8d Created pod: workspace7722130dbd9049ee-57c6b44b8d-tkq95
0s Normal Scheduled pod/workspace7722130dbd9049ee-57c6b44b8d-tkq95 Successfully assigned divine-chelladurai-che-pmjg9u/workspace7722130dbd9049ee-57c6b44b8d-tkq95 to tkc-workers-wnnlk-b55cc6bdd-d6767
0s Normal SuccessfulDelete replicaset/workspace7722130dbd9049ee-57c6b44b8d Deleted pod: workspace7722130dbd9049ee-57c6b44b8d-tkq95
0s Warning FailedAttachVolume pod/workspace7722130dbd9049ee-5f5fbbc678-s9r62 AttachVolume.Attach failed for volume "pvc-80278a68-7f9a-4e24-83b5-ff4fdf993a35" : rpc error: code = Aborted desc = pending
0s Warning FailedMount pod/workspace7722130dbd9049ee-5f5fbbc678-t5k5j Unable to attach or mount volumes: unmounted volumes=[kube-api-access-bpt4f che-gateway claim-devworkspace automount-secret-devworkspace-merged-git-credentials workspace-metadata automount-configmap-che-trusted-ca-certs automount-secret-user-profile remote-endpoint automount-configmap-devworkspace-gitconfig automount-secret-che-server-cert], unattached volumes=[kube-api-access-bpt4f che-gateway claim-devworkspace automount-secret-devworkspace-merged-git-credentials workspace-metadata automount-configmap-che-trusted-ca-certs automount-secret-user-profile remote-endpoint automount-configmap-devworkspace-gitconfig automount-secret-che-server-cert]: timed out waiting for the condition
0s Warning FailedMount pod/workspace7722130dbd9049ee-5f5fbbc678-brt29 Unable to attach or mount volumes: unmounted volumes=[automount-configmap-che-trusted-ca-certs automount-secret-che-server-cert automount-secret-user-profile kube-api-access-496j7 che-gateway automount-configmap-devworkspace-gitconfig workspace-metadata claim-devworkspace automount-secret-devworkspace-merged-git-credentials remote-endpoint], unattached volumes=[automount-configmap-che-trusted-ca-certs automount-secret-che-server-cert automount-secret-user-profile kube-api-access-496j7 che-gateway automount-configmap-devworkspace-gitconfig workspace-metadata claim-devworkspace automount-secret-devworkspace-merged-git-credentials remote-endpoint]: timed out waiting for the condition
0s Warning FailedMount pod/workspace7722130dbd9049ee-5f5fbbc678-dzz9r Unable to attach or mount volumes: unmounted volumes=[automount-configmap-devworkspace-gitconfig claim-devworkspace automount-secret-devworkspace-merged-git-credentials automount-secret-che-server-cert automount-secret-user-profile che-gateway remote-endpoint kube-api-access-mvgj5 automount-configmap-che-trusted-ca-certs workspace-metadata], unattached volumes=[automount-configmap-devworkspace-gitconfig claim-devworkspace automount-secret-devworkspace-merged-git-credentials automount-secret-che-server-cert automount-secret-user-profile che-gateway remote-endpoint kube-api-access-mvgj5 automount-configmap-che-trusted-ca-certs workspace-metadata]: timed out waiting for the condition
0s Warning FailedMount pod/workspace7722130dbd9049ee-5f5fbbc678-pj97z Unable to attach or mount volumes: unmounted volumes=[che-gateway automount-configmap-devworkspace-gitconfig kube-api-access-9cnc9 remote-endpoint automount-secret-user-profile claim-devworkspace automount-secret-devworkspace-merged-git-credentials automount-configmap-che-trusted-ca-certs automount-secret-che-server-cert workspace-metadata], unattached volumes=[che-gateway automount-configmap-devworkspace-gitconfig kube-api-access-9cnc9 remote-endpoint automount-secret-user-profile claim-devworkspace automount-secret-devworkspace-merged-git-credentials automount-configmap-che-trusted-ca-certs automount-secret-che-server-cert workspace-metadata]: timed out waiting for the condition
0s Warning FailedMount pod/workspace7722130dbd9049ee-5f5fbbc678-s9r62 Unable to attach or mount volumes: unmounted volumes=[automount-secret-che-server-cert workspace-metadata automount-configmap-devworkspace-gitconfig automount-secret-user-profile kube-api-access-55lhw remote-endpoint automount-secret-devworkspace-merged-git-credentials automount-configmap-che-trusted-ca-certs che-gateway claim-devworkspace], unattached volumes=[automount-secret-che-server-cert workspace-metadata automount-configmap-devworkspace-gitconfig automount-secret-user-profile kube-api-access-55lhw remote-endpoint automount-secret-devworkspace-merged-git-credentials automount-configmap-che-trusted-ca-certs che-gateway claim-devworkspace]: timed out waiting for the condition
0s Warning FailedMount pod/workspace7722130dbd9049ee-57c6b44b8d-tkq95 Unable to attach or mount volumes: unmounted volumes=[workspace-metadata kube-api-access-hchw5 automount-secret-devworkspace-merged-git-credentials automount-configmap-devworkspace-gitconfig automount-configmap-che-trusted-ca-certs automount-secret-user-profile automount-secret-che-server-cert claim-devworkspace che-gateway remote-endpoint], unattached volumes=[workspace-metadata kube-api-access-hchw5 automount-secret-devworkspace-merged-git-credentials automount-configmap-devworkspace-gitconfig automount-configmap-che-trusted-ca-certs automount-secret-user-profile automount-secret-che-server-cert claim-devworkspace che-gateway remote-endpoint]: timed out waiting for the condition
below is my devfile.yaml. i'm not sure how to analyze the root cause for this issue. please help me out on this.
schemaVersion: 2.1.0
metadata:
name: cbfsel-repo
projects:
- name: cbfsel-project
git:
checkoutFrom:
revision: master
remotes:
origin: https://gitlab.com/dchelladurai/cbf-sel.git
components:
- container:
image: 'artfact-prd/qedocker/eclipseche/customopenjdk8:v10'
memoryLimit: 4G
name: javacontainer
commands:
- exec:
commandLine: mvn clean package -DskipTests
component: javacontainer
group:
isDefault: true
kind: build
label: 'build project using maven'
workingDir: '${PROJECT_SOURCE}'
id: mvnpackage
below link has details on how pvc is used by workspace, i'm just confused on why this error appears for my setup. i have pv,pvc and storageclass available in my cluster. please help me to fix this issue https://www.eclipse.org/che/docs/stable/end-user-guide/requesting-persistent-storage-for-workspaces/
Relevant information
No response
Failure log taken at a later time
dchelladura-a01:vtapbotservice divine$ kubectl get events -w -n divine-chelladurai-che-pmjg9u
LAST SEEN TYPE REASON OBJECT MESSAGE
17s Normal Scheduled pod/workspace7722130dbd9049ee-57c6b44b8d-cv6c4 Successfully assigned divine-chelladurai-che-pmjg9u/workspace7722130dbd9049ee-57c6b44b8d-cv6c4 to tkc-workers-wnnlk-b55cc6bdd-d6767
14s Normal SuccessfulAttachVolume pod/workspace7722130dbd9049ee-57c6b44b8d-cv6c4 AttachVolume.Attach succeeded for volume "pvc-80278a68-7f9a-4e24-83b5-ff4fdf993a35"
18s Normal SuccessfulCreate replicaset/workspace7722130dbd9049ee-57c6b44b8d Created pod: workspace7722130dbd9049ee-57c6b44b8d-cv6c4
18s Normal ScalingReplicaSet deployment/workspace7722130dbd9049ee Scaled up replica set workspace7722130dbd9049ee-57c6b44b8d to 1
0s Normal ScalingReplicaSet deployment/workspace7722130dbd9049ee Scaled down replica set workspace7722130dbd9049ee-57c6b44b8d to 0
0s Normal SuccessfulDelete replicaset/workspace7722130dbd9049ee-57c6b44b8d Deleted pod: workspace7722130dbd9049ee-57c6b44b8d-cv6c4
0s Normal ScalingReplicaSet deployment/workspace7722130dbd9049ee Scaled up replica set workspace7722130dbd9049ee-5f5fbbc678 to 1
0s Normal SuccessfulCreate replicaset/workspace7722130dbd9049ee-5f5fbbc678 Created pod: workspace7722130dbd9049ee-5f5fbbc678-rpll4
0s Normal Scheduled pod/workspace7722130dbd9049ee-5f5fbbc678-rpll4 Successfully assigned divine-chelladurai-che-pmjg9u/workspace7722130dbd9049ee-5f5fbbc678-rpll4 to tkc-workers-wnnlk-b55cc6bdd-d6767
0s Normal SuccessfulDelete replicaset/workspace7722130dbd9049ee-5f5fbbc678 Deleted pod: workspace7722130dbd9049ee-5f5fbbc678-rpll4
0s Normal ScalingReplicaSet deployment/workspace7722130dbd9049ee Scaled down replica set workspace7722130dbd9049ee-5f5fbbc678 to 0
0s Normal ScalingReplicaSet deployment/workspace7722130dbd9049ee Scaled up replica set workspace7722130dbd9049ee-5f5fbbc678 to 1
0s Normal Scheduled pod/workspace7722130dbd9049ee-5f5fbbc678-lnqbh Successfully assigned divine-chelladurai-che-pmjg9u/workspace7722130dbd9049ee-5f5fbbc678-lnqbh to tkc-workers-wnnlk-b55cc6bdd-h75s7
0s Normal SuccessfulCreate replicaset/workspace7722130dbd9049ee-5f5fbbc678 Created pod: workspace7722130dbd9049ee-5f5fbbc678-lnqbh
0s Warning FailedAttachVolume pod/workspace7722130dbd9049ee-5f5fbbc678-lnqbh Multi-Attach error for volume "pvc-80278a68-7f9a-4e24-83b5-ff4fdf993a35" Volume is already exclusively attached to one node and can't be attached to another
0s Normal ScalingReplicaSet deployment/workspace7722130dbd9049ee Scaled down replica set workspace7722130dbd9049ee-5f5fbbc678 to 0
0s Normal SuccessfulDelete replicaset/workspace7722130dbd9049ee-5f5fbbc678 Deleted pod: workspace7722130dbd9049ee-5f5fbbc678-lnqbh
0s Normal ScalingReplicaSet deployment/workspace7722130dbd9049ee Scaled up replica set workspace7722130dbd9049ee-5f5fbbc678 to 1
0s Normal SuccessfulCreate replicaset/workspace7722130dbd9049ee-5f5fbbc678 Created pod: workspace7722130dbd9049ee-5f5fbbc678-hnbh2
0s Normal Scheduled pod/workspace7722130dbd9049ee-5f5fbbc678-hnbh2 Successfully assigned divine-chelladurai-che-pmjg9u/workspace7722130dbd9049ee-5f5fbbc678-hnbh2 to tkc-workers-wnnlk-b55cc6bdd-7k767
0s Warning FailedAttachVolume pod/workspace7722130dbd9049ee-5f5fbbc678-hnbh2 Multi-Attach error for volume "pvc-80278a68-7f9a-4e24-83b5-ff4fdf993a35" Volume is already exclusively attached to one node and can't be attached to another
0s Warning FailedMount pod/workspace7722130dbd9049ee-57c6b44b8d-cv6c4 Unable to attach or mount volumes: unmounted volumes=[remote-endpoint automount-configmap-devworkspace-gitconfig automount-secret-user-profile che-gateway workspace-metadata claim-devworkspace automount-secret-devworkspace-merged-git-credentials automount-configmap-che-trusted-ca-certs automount-secret-che-server-cert kube-api-access-4gh9w], unattached volumes=[remote-endpoint automount-configmap-devworkspace-gitconfig automount-secret-user-profile che-gateway workspace-metadata claim-devworkspace automount-secret-devworkspace-merged-git-credentials automount-configmap-che-trusted-ca-certs automount-secret-che-server-cert kube-api-access-4gh9w]: timed out waiting for the condition
0s Normal ScalingReplicaSet deployment/workspace7722130dbd9049ee Scaled down replica set workspace7722130dbd9049ee-5f5fbbc678 to 0
0s Normal SuccessfulDelete replicaset/workspace7722130dbd9049ee-5f5fbbc678 Deleted pod: workspace7722130dbd9049ee-5f5fbbc678-hnbh2
0s Normal ScalingReplicaSet deployment/workspace7722130dbd9049ee Scaled up replica set workspace7722130dbd9049ee-5f5fbbc678 to 1
0s Normal SuccessfulCreate replicaset/workspace7722130dbd9049ee-5f5fbbc678 Created pod: workspace7722130dbd9049ee-5f5fbbc678-fblpl
1s Normal Scheduled pod/workspace7722130dbd9049ee-5f5fbbc678-fblpl Successfully assigned divine-chelladurai-che-pmjg9u/workspace7722130dbd9049ee-5f5fbbc678-fblpl to tkc-workers-wnnlk-b55cc6bdd-d6767
0s Normal ScalingReplicaSet deployment/workspace7722130dbd9049ee Scaled down replica set workspace7722130dbd9049ee-5f5fbbc678 to 0
0s Normal SuccessfulDelete replicaset/workspace7722130dbd9049ee-5f5fbbc678 Deleted pod: workspace7722130dbd9049ee-5f5fbbc678-fblpl
0s Normal ScalingReplicaSet deployment/workspace7722130dbd9049ee Scaled up replica set workspace7722130dbd9049ee-5f5fbbc678 to 1
0s Normal SuccessfulCreate replicaset/workspace7722130dbd9049ee-5f5fbbc678 Created pod: workspace7722130dbd9049ee-5f5fbbc678-2x5wt
0s Normal Scheduled pod/workspace7722130dbd9049ee-5f5fbbc678-2x5wt Successfully assigned divine-chelladurai-che-pmjg9u/workspace7722130dbd9049ee-5f5fbbc678-2x5wt to tkc-workers-wnnlk-b55cc6bdd-d6767
0s Normal ScalingReplicaSet deployment/workspace7722130dbd9049ee Scaled down replica set workspace7722130dbd9049ee-5f5fbbc678 to 0
0s Normal SuccessfulDelete replicaset/workspace7722130dbd9049ee-5f5fbbc678 Deleted pod: workspace7722130dbd9049ee-5f5fbbc678-2x5wt
0s Normal ScalingReplicaSet deployment/workspace7722130dbd9049ee Scaled up replica set workspace7722130dbd9049ee-5f5fbbc678 to 1
0s Normal SuccessfulCreate replicaset/workspace7722130dbd9049ee-5f5fbbc678 Created pod: workspace7722130dbd9049ee-5f5fbbc678-gpbw5
0s Normal Scheduled pod/workspace7722130dbd9049ee-5f5fbbc678-gpbw5 Successfully assigned divine-chelladurai-che-pmjg9u/workspace7722130dbd9049ee-5f5fbbc678-gpbw5 to tkc-workers-wnnlk-b55cc6bdd-h75s7
0s Normal SuccessfulDelete replicaset/workspace7722130dbd9049ee-5f5fbbc678 Deleted pod: workspace7722130dbd9049ee-5f5fbbc678-gpbw5
0s Normal ScalingReplicaSet deployment/workspace7722130dbd9049ee Scaled down replica set workspace7722130dbd9049ee-5f5fbbc678 to 0
0s Normal ScalingReplicaSet deployment/workspace7722130dbd9049ee Scaled up replica set workspace7722130dbd9049ee-75d48cd9b4 to 1
0s Normal SuccessfulCreate replicaset/workspace7722130dbd9049ee-75d48cd9b4 Created pod: workspace7722130dbd9049ee-75d48cd9b4-qz2mn
0s Normal Scheduled pod/workspace7722130dbd9049ee-75d48cd9b4-qz2mn Successfully assigned divine-chelladurai-che-pmjg9u/workspace7722130dbd9049ee-75d48cd9b4-qz2mn to tkc-workers-wnnlk-b55cc6bdd-d6767
0s Normal ScalingReplicaSet deployment/workspace7722130dbd9049ee Scaled down replica set workspace7722130dbd9049ee-75d48cd9b4 to 0
0s Normal SuccessfulDelete replicaset/workspace7722130dbd9049ee-75d48cd9b4 Deleted pod: workspace7722130dbd9049ee-75d48cd9b4-qz2mn
0s Normal ScalingReplicaSet deployment/workspace7722130dbd9049ee Scaled up replica set workspace7722130dbd9049ee-5f5fbbc678 to 1
0s Normal SuccessfulCreate replicaset/workspace7722130dbd9049ee-5f5fbbc678 Created pod: workspace7722130dbd9049ee-5f5fbbc678-f6p7q
0s Normal Scheduled pod/workspace7722130dbd9049ee-5f5fbbc678-f6p7q Successfully assigned divine-chelladurai-che-pmjg9u/workspace7722130dbd9049ee-5f5fbbc678-f6p7q to tkc-workers-wnnlk-b55cc6bdd-d6767
0s Normal ScalingReplicaSet deployment/workspace7722130dbd9049ee Scaled down replica set workspace7722130dbd9049ee-5f5fbbc678 to 0
0s Normal SuccessfulDelete replicaset/workspace7722130dbd9049ee-5f5fbbc678 Deleted pod: workspace7722130dbd9049ee-5f5fbbc678-f6p7q
0s Normal ScalingReplicaSet deployment/workspace7722130dbd9049ee Scaled up replica set workspace7722130dbd9049ee-5f5fbbc678 to 1
0s Normal SuccessfulCreate replicaset/workspace7722130dbd9049ee-5f5fbbc678 Created pod: workspace7722130dbd9049ee-5f5fbbc678-7hxmq
0s Normal Scheduled pod/workspace7722130dbd9049ee-5f5fbbc678-7hxmq Successfully assigned divine-chelladurai-che-pmjg9u/workspace7722130dbd9049ee-5f5fbbc678-7hxmq to tkc-workers-wnnlk-b55cc6bdd-7k767
0s Normal ScalingReplicaSet deployment/workspace7722130dbd9049ee Scaled down replica set workspace7722130dbd9049ee-5f5fbbc678 to 0
0s Normal SuccessfulDelete replicaset/workspace7722130dbd9049ee-5f5fbbc678 Deleted pod: workspace7722130dbd9049ee-5f5fbbc678-7hxmq
0s Normal ScalingReplicaSet deployment/workspace7722130dbd9049ee Scaled up replica set workspace7722130dbd9049ee-5f5fbbc678 to 1
0s Normal SuccessfulCreate replicaset/workspace7722130dbd9049ee-5f5fbbc678 Created pod: workspace7722130dbd9049ee-5f5fbbc678-hxbkf
0s Normal Scheduled pod/workspace7722130dbd9049ee-5f5fbbc678-hxbkf Successfully assigned divine-chelladurai-che-pmjg9u/workspace7722130dbd9049ee-5f5fbbc678-hxbkf to tkc-workers-wnnlk-b55cc6bdd-h75s7
0s Warning FailedMount pod/workspace7722130dbd9049ee-5f5fbbc678-7hxmq MountVolume.SetUp failed for volume "automount-secret-devworkspace-merged-git-credentials" : failed to sync secret cache: timed out waiting for the condition
0s Normal ScalingReplicaSet deployment/workspace7722130dbd9049ee Scaled down replica set workspace7722130dbd9049ee-5f5fbbc678 to 0
0s Normal SuccessfulDelete replicaset/workspace7722130dbd9049ee-5f5fbbc678 Deleted pod: workspace7722130dbd9049ee-5f5fbbc678-hxbkf
0s Normal ScalingReplicaSet deployment/workspace7722130dbd9049ee Scaled up replica set workspace7722130dbd9049ee-5f5fbbc678 to 1
0s Normal Scheduled pod/workspace7722130dbd9049ee-5f5fbbc678-mk4qs Successfully assigned divine-chelladurai-che-pmjg9u/workspace7722130dbd9049ee-5f5fbbc678-mk4qs to tkc-workers-wnnlk-b55cc6bdd-d6767
0s Normal SuccessfulCreate replicaset/workspace7722130dbd9049ee-5f5fbbc678 (combined from similar events): Created pod: workspace7722130dbd9049ee-5f5fbbc678-mk4qs
0s Warning FailedMount pod/workspace7722130dbd9049ee-5f5fbbc678-rpll4 Unable to attach or mount volumes: unmounted volumes=[workspace-metadata claim-devworkspace che-gateway automount-configmap-devworkspace-gitconfig kube-api-access-r5795 automount-secret-devworkspace-merged-git-credentials automount-secret-user-profile automount-secret-che-server-cert remote-endpoint automount-configmap-che-trusted-ca-certs], unattached volumes=[workspace-metadata claim-devworkspace che-gateway automount-configmap-devworkspace-gitconfig kube-api-access-r5795 automount-secret-devworkspace-merged-git-credentials automount-secret-user-profile automount-secret-che-server-cert remote-endpoint automount-configmap-che-trusted-ca-certs]: timed out waiting for the condition
0s Normal ScalingReplicaSet deployment/workspace7722130dbd9049ee Scaled down replica set workspace7722130dbd9049ee-5f5fbbc678 to 0
0s Normal SuccessfulDelete replicaset/workspace7722130dbd9049ee-5f5fbbc678 (combined from similar events): Deleted pod: workspace7722130dbd9049ee-5f5fbbc678-mk4qs
0s Normal ScalingReplicaSet deployment/workspace7722130dbd9049ee Scaled up replica set workspace7722130dbd9049ee-5f5fbbc678 to 1
0s Normal SuccessfulCreate replicaset/workspace7722130dbd9049ee-5f5fbbc678 (combined from similar events): Created pod: workspace7722130dbd9049ee-5f5fbbc678-nfspg
0s Normal Scheduled pod/workspace7722130dbd9049ee-5f5fbbc678-nfspg Successfully assigned divine-chelladurai-che-pmjg9u/workspace7722130dbd9049ee-5f5fbbc678-nfspg to tkc-workers-wnnlk-b55cc6bdd-7k767
0s Normal SuccessfulDelete replicaset/workspace7722130dbd9049ee-5f5fbbc678 (combined from similar events): Deleted pod: workspace7722130dbd9049ee-5f5fbbc678-nfspg
0s Normal Scheduled pod/workspace7722130dbd9049ee-5f5fbbc678-6fhlg Successfully assigned divine-chelladurai-che-pmjg9u/workspace7722130dbd9049ee-5f5fbbc678-6fhlg to tkc-workers-wnnlk-b55cc6bdd-d6767
0s Normal SuccessfulCreate replicaset/workspace7722130dbd9049ee-5f5fbbc678 (combined from similar events): Created pod: workspace7722130dbd9049ee-5f5fbbc678-6fhlg
0s Warning FailedMount pod/workspace7722130dbd9049ee-5f5fbbc678-lnqbh Unable to attach or mount volumes: unmounted volumes=[automount-secret-user-profile workspace-metadata che-gateway automount-configmap-che-trusted-ca-certs claim-devworkspace automount-secret-devworkspace-merged-git-credentials automount-secret-che-server-cert kube-api-access-k59kj remote-endpoint automount-configmap-devworkspace-gitconfig], unattached volumes=[automount-secret-user-profile workspace-metadata che-gateway automount-configmap-che-trusted-ca-certs claim-devworkspace automount-secret-devworkspace-merged-git-credentials automount-secret-che-server-cert kube-api-access-k59kj remote-endpoint automount-configmap-devworkspace-gitconfig]: timed out waiting for the condition
0s Warning FailedMount pod/workspace7722130dbd9049ee-5f5fbbc678-hnbh2 Unable to attach or mount volumes: unmounted volumes=[automount-configmap-devworkspace-gitconfig automount-configmap-che-trusted-ca-certs che-gateway automount-secret-user-profile kube-api-access-kbqqf remote-endpoint automount-secret-devworkspace-merged-git-credentials workspace-metadata claim-devworkspace automount-secret-che-server-cert], unattached volumes=[automount-configmap-devworkspace-gitconfig automount-configmap-che-trusted-ca-certs che-gateway automount-secret-user-profile kube-api-access-kbqqf remote-endpoint automount-secret-devworkspace-merged-git-credentials workspace-metadata claim-devworkspace automount-secret-che-server-cert]: timed out waiting for the condition
0s Warning FailedMount pod/workspace7722130dbd9049ee-5f5fbbc678-6fhlg MountVolume.SetUp failed for volume "automount-configmap-che-trusted-ca-certs" : failed to sync configmap cache: timed out waiting for the condition
0s Warning FailedMount pod/workspace7722130dbd9049ee-5f5fbbc678-6fhlg MountVolume.SetUp failed for volume "automount-secret-user-profile" : failed to sync secret cache: timed out waiting for the condition
0s Warning FailedMount pod/workspace7722130dbd9049ee-5f5fbbc678-6fhlg MountVolume.SetUp failed for volume "automount-secret-devworkspace-merged-git-credentials" : failed to sync secret cache: timed out waiting for the condition
0s Warning FailedMount pod/workspace7722130dbd9049ee-5f5fbbc678-6fhlg MountVolume.SetUp failed for volume "automount-configmap-devworkspace-gitconfig" : failed to sync configmap cache: timed out waiting for the condition
0s Warning FailedMount pod/workspace7722130dbd9049ee-5f5fbbc678-6fhlg MountVolume.SetUp failed for volume "automount-secret-che-server-cert" : failed to sync secret cache: timed out waiting for the condition
0s Warning FailedMount pod/workspace7722130dbd9049ee-5f5fbbc678-6fhlg MountVolume.SetUp failed for volume "che-gateway" : failed to sync configmap cache: timed out waiting for the condition
0s Warning FailedMount pod/workspace7722130dbd9049ee-5f5fbbc678-6fhlg MountVolume.SetUp failed for volume "workspace-metadata" : failed to sync configmap cache: timed out waiting for the condition
0s Warning FailedMount pod/workspace7722130dbd9049ee-5f5fbbc678-6fhlg MountVolume.SetUp failed for volume "automount-secret-devworkspace-merged-git-credentials" : failed to sync secret cache: timed out waiting for the condition
0s Warning FailedAttachVolume pod/workspace7722130dbd9049ee-5f5fbbc678-hnbh2 AttachVolume.Attach failed for volume "pvc-80278a68-7f9a-4e24-83b5-ff4fdf993a35" : Attach timeout for volume df63f994-eaeb-47a8-ac23-0e222cac3b10-80278a68-7f9a-4e24-83b5-ff4fdf993a35
0s Warning FailedMount pod/workspace7722130dbd9049ee-5f5fbbc678-fblpl Unable to attach or mount volumes: unmounted volumes=[automount-configmap-che-trusted-ca-certs automount-secret-user-profile claim-devworkspace remote-endpoint automount-configmap-devworkspace-gitconfig che-gateway workspace-metadata automount-secret-devworkspace-merged-git-credentials automount-secret-che-server-cert kube-api-access-2j4k6], unattached volumes=[automount-configmap-che-trusted-ca-certs automount-secret-user-profile claim-devworkspace remote-endpoint automount-configmap-devworkspace-gitconfig che-gateway workspace-metadata automount-secret-devworkspace-merged-git-credentials automount-secret-che-server-cert kube-api-access-2j4k6]: timed out waiting for the condition
0s Warning FailedMount pod/workspace7722130dbd9049ee-5f5fbbc678-2x5wt Unable to attach or mount volumes: unmounted volumes=[automount-secret-devworkspace-merged-git-credentials workspace-metadata automount-secret-che-server-cert kube-api-access-p2gp8 automount-secret-user-profile automount-configmap-devworkspace-gitconfig automount-configmap-che-trusted-ca-certs che-gateway claim-devworkspace remote-endpoint], unattached volumes=[automount-secret-devworkspace-merged-git-credentials workspace-metadata automount-secret-che-server-cert kube-api-access-p2gp8 automount-secret-user-profile automount-configmap-devworkspace-gitconfig automount-configmap-che-trusted-ca-certs che-gateway claim-devworkspace remote-endpoint]: timed out waiting for the condition
0s Warning FailedMount pod/workspace7722130dbd9049ee-5f5fbbc678-gpbw5 Unable to attach or mount volumes: unmounted volumes=[automount-secret-che-server-cert automount-secret-devworkspace-merged-git-credentials automount-configmap-devworkspace-gitconfig kube-api-access-w4f52 che-gateway remote-endpoint automount-configmap-che-trusted-ca-certs automount-secret-user-profile workspace-metadata claim-devworkspace], unattached volumes=[automount-secret-che-server-cert automount-secret-devworkspace-merged-git-credentials automount-configmap-devworkspace-gitconfig kube-api-access-w4f52 che-gateway remote-endpoint automount-configmap-che-trusted-ca-certs automount-secret-user-profile workspace-metadata claim-devworkspace]: timed out waiting for the condition
0s Warning FailedMount pod/workspace7722130dbd9049ee-75d48cd9b4-qz2mn Unable to attach or mount volumes: unmounted volumes=[remote-endpoint automount-configmap-devworkspace-gitconfig automount-secret-user-profile workspace-metadata claim-devworkspace automount-configmap-che-trusted-ca-certs kube-api-access-2g9x6 che-gateway automount-secret-devworkspace-merged-git-credentials automount-secret-che-server-cert], unattached volumes=[remote-endpoint automount-configmap-devworkspace-gitconfig automount-secret-user-profile workspace-metadata claim-devworkspace automount-configmap-che-trusted-ca-certs kube-api-access-2g9x6 che-gateway automount-secret-devworkspace-merged-git-credentials automount-secret-che-server-cert]: timed out waiting for the condition
0s Warning FailedMount pod/workspace7722130dbd9049ee-5f5fbbc678-f6p7q Unable to attach or mount volumes: unmounted volumes=[automount-configmap-che-trusted-ca-certs automount-secret-che-server-cert automount-secret-devworkspace-merged-git-credentials workspace-metadata kube-api-access-wzcqb che-gateway claim-devworkspace remote-endpoint automount-configmap-devworkspace-gitconfig automount-secret-user-profile], unattached volumes=[automount-configmap-che-trusted-ca-certs automount-secret-che-server-cert automount-secret-devworkspace-merged-git-credentials workspace-metadata kube-api-access-wzcqb che-gateway claim-devworkspace remote-endpoint automount-configmap-devworkspace-gitconfig automount-secret-user-profile]: timed out waiting for the condition
0s Warning FailedMount pod/workspace7722130dbd9049ee-5f5fbbc678-7hxmq Unable to attach or mount volumes: unmounted volumes=[automount-configmap-devworkspace-gitconfig kube-api-access-z76wm che-gateway automount-secret-che-server-cert automount-secret-devworkspace-merged-git-credentials claim-devworkspace automount-secret-user-profile workspace-metadata remote-endpoint automount-configmap-che-trusted-ca-certs], unattached volumes=[automount-configmap-devworkspace-gitconfig kube-api-access-z76wm che-gateway automount-secret-che-server-cert automount-secret-devworkspace-merged-git-credentials claim-devworkspace automount-secret-user-profile workspace-metadata remote-endpoint automount-configmap-che-trusted-ca-certs]: timed out waiting for the condition
0s Warning FailedMount pod/workspace7722130dbd9049ee-5f5fbbc678-hxbkf Unable to attach or mount volumes: unmounted volumes=[automount-secret-user-profile claim-devworkspace remote-endpoint automount-secret-che-server-cert che-gateway kube-api-access-5mbnp workspace-metadata automount-secret-devworkspace-merged-git-credentials automount-configmap-devworkspace-gitconfig automount-configmap-che-trusted-ca-certs], unattached volumes=[automount-secret-user-profile claim-devworkspace remote-endpoint automount-secret-che-server-cert che-gateway kube-api-access-5mbnp workspace-metadata automount-secret-devworkspace-merged-git-credentials automount-configmap-devworkspace-gitconfig automount-configmap-che-trusted-ca-certs]: timed out waiting for the condition
0s Normal SuccessfulDelete replicaset/workspace7722130dbd9049ee-5f5fbbc678 (combined from similar events): Deleted pod: workspace7722130dbd9049ee-5f5fbbc678-6fhlg
0
captured another failure log
dchelladura-a01:vtapbotservice divine$ kubectl get events -w -n divine-chelladurai-che-pmjg9u
LAST SEEN TYPE REASON OBJECT MESSAGE
15s Normal Provisioning persistentvolumeclaim/claim-devworkspace External provisioner is provisioning volume for claim "divine-chelladurai-che-pmjg9u/claim-devworkspace"
15s Normal ExternalProvisioning persistentvolumeclaim/claim-devworkspace waiting for a volume to be created, either by external provisioner "csi.vsphere.vmware.com" or manually created by system administrator
13s Normal ProvisioningSucceeded persistentvolumeclaim/claim-devworkspace Successfully provisioned volume pvc-65ab8953-6926-4c53-b0d2-b9ebb2153b74
14s Warning FailedScheduling pod/workspace027a7a2667034873-ff6dff45b-zvbhm 0/4 nodes are available: 4 pod has unbound immediate PersistentVolumeClaims.
14s Normal SuccessfulCreate replicaset/workspace027a7a2667034873-ff6dff45b Created pod: workspace027a7a2667034873-ff6dff45b-zvbhm
14s Normal SuccessfulDelete replicaset/workspace027a7a2667034873-ff6dff45b Deleted pod: workspace027a7a2667034873-ff6dff45b-zvbhm
14s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled up replica set workspace027a7a2667034873-ff6dff45b to 1
14s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled down replica set workspace027a7a2667034873-ff6dff45b to 0
i tried to start the same workspace today morning . the workspace started successfully.
does this error occur due to some issue in my infrastructure? does this error occur due to unavailable volumes in my infrastructure? if yes, please help me understand how to identify the exact error, so that i can communicate the proof to my infrastructure team.
dchelladura-a01:vtapbotservice divine$ kubectl get events -w -n divine-chelladurai-che-pmjg9u
LAST SEEN TYPE REASON OBJECT MESSAGE
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled up replica set workspace027a7a2667034873-ff6dff45b to 1
0s Normal SuccessfulCreate replicaset/workspace027a7a2667034873-ff6dff45b Created pod: workspace027a7a2667034873-ff6dff45b-rh2mh
0s Normal Scheduled pod/workspace027a7a2667034873-ff6dff45b-rh2mh Successfully assigned divine-chelladurai-che-pmjg9u/workspace027a7a2667034873-ff6dff45b-rh2mh to tkc-workers-wnnlk-b55cc6bdd-d6767
0s Normal SuccessfulAttachVolume pod/workspace027a7a2667034873-ff6dff45b-rh2mh AttachVolume.Attach succeeded for volume "pvc-65ab8953-6926-4c53-b0d2-b9ebb2153b74"
0s Normal Pulling pod/workspace027a7a2667034873-ff6dff45b-rh2mh Pulling image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next"
0s Normal Pulled pod/workspace027a7a2667034873-ff6dff45b-rh2mh Successfully pulled image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next" in 412.361465ms
0s Normal Created pod/workspace027a7a2667034873-ff6dff45b-rh2mh Created container remote-runtime-injector
0s Normal Started pod/workspace027a7a2667034873-ff6dff45b-rh2mh Started container remote-runtime-injector
0s Normal Pulling pod/workspace027a7a2667034873-ff6dff45b-rh2mh Pulling image "quay.io/devfile/project-clone:v0.15.2"
0s Normal Pulled pod/workspace027a7a2667034873-ff6dff45b-rh2mh Successfully pulled image "quay.io/devfile/project-clone:v0.15.2" in 342.432791ms
0s Normal Created pod/workspace027a7a2667034873-ff6dff45b-rh2mh Created container project-clone
0s Normal Started pod/workspace027a7a2667034873-ff6dff45b-rh2mh Started container project-clone
0s Normal Pulling pod/workspace027a7a2667034873-ff6dff45b-rh2mh Pulling image "quay.io/eclipse/che-theia:next"
0s Normal Pulled pod/workspace027a7a2667034873-ff6dff45b-rh2mh Successfully pulled image "quay.io/eclipse/che-theia:next" in 337.606743ms
0s Normal Created pod/workspace027a7a2667034873-ff6dff45b-rh2mh Created container theia-ide
0s Normal Started pod/workspace027a7a2667034873-ff6dff45b-rh2mh Started container theia-ide
0s Normal Pulling pod/workspace027a7a2667034873-ff6dff45b-rh2mh Pulling image "quay.io/eclipse/che-machine-exec:next"
0s Normal Pulled pod/workspace027a7a2667034873-ff6dff45b-rh2mh Successfully pulled image "quay.io/eclipse/che-machine-exec:next" in 350.363102ms
0s Normal Created pod/workspace027a7a2667034873-ff6dff45b-rh2mh Created container che-machine-exec
0s Normal Started pod/workspace027a7a2667034873-ff6dff45b-rh2mh Started container che-machine-exec
0s Normal Pulling pod/workspace027a7a2667034873-ff6dff45b-rh2mh Pulling image "artfact-prd.vmware.com:5001/qedocker/eclipseche/customopenjdk8:v10"
0s Normal Pulled pod/workspace027a7a2667034873-ff6dff45b-rh2mh Successfully pulled image "artfact-prd.vmware.com:5001/qedocker/eclipseche/customopenjdk8:v10" in 370.811363ms
0s Normal Created pod/workspace027a7a2667034873-ff6dff45b-rh2mh Created container javacontainer
0s Normal Started pod/workspace027a7a2667034873-ff6dff45b-rh2mh Started container javacontainer
0s Normal Pulled pod/workspace027a7a2667034873-ff6dff45b-rh2mh Container image "quay.io/eclipse/che--traefik:v2.8.1-4e52a5e2495484f5e19a49edfd2f652b0bce7b3603fa0df545ed90168ffae1c3" already present on machine
0s Normal Created pod/workspace027a7a2667034873-ff6dff45b-rh2mh Created container che-gateway
0s Normal Started pod/workspace027a7a2667034873-ff6dff45b-rh2mh Started container che-gateway
please help me understand how to identify the rootcause of this issue, so that i can fix this issue.
i restarted my workspace as i installed a plugin in the workspace. now my workspace is not starting due to below error
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled down replica set workspace027a7a2667034873-ff6dff45b to 0
0s Normal Killing pod/workspace027a7a2667034873-ff6dff45b-rh2mh Stopping container theia-ide
0s Normal SuccessfulDelete replicaset/workspace027a7a2667034873-ff6dff45b Deleted pod: workspace027a7a2667034873-ff6dff45b-rh2mh
0s Normal Killing pod/workspace027a7a2667034873-ff6dff45b-rh2mh Stopping container che-gateway
0s Normal Killing pod/workspace027a7a2667034873-ff6dff45b-rh2mh Stopping container che-machine-exec
0s Normal Killing pod/workspace027a7a2667034873-ff6dff45b-rh2mh Stopping container javacontainer
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled up replica set workspace027a7a2667034873-ff6dff45b to 1
0s Normal SuccessfulCreate replicaset/workspace027a7a2667034873-ff6dff45b Created pod: workspace027a7a2667034873-ff6dff45b-lkmmp
0s Normal Scheduled pod/workspace027a7a2667034873-ff6dff45b-lkmmp Successfully assigned divine-chelladurai-che-pmjg9u/workspace027a7a2667034873-ff6dff45b-lkmmp to tkc-workers-wnnlk-b55cc6bdd-7k767
0s Warning FailedAttachVolume pod/workspace027a7a2667034873-ff6dff45b-lkmmp Multi-Attach error for volume "pvc-65ab8953-6926-4c53-b0d2-b9ebb2153b74" Volume is already exclusively attached to one node and can't be attached to another
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled down replica set workspace027a7a2667034873-ff6dff45b to 0
0s Normal SuccessfulDelete replicaset/workspace027a7a2667034873-ff6dff45b Deleted pod: workspace027a7a2667034873-ff6dff45b-lkmmp
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled up replica set workspace027a7a2667034873-ff6dff45b to 1
0s Normal SuccessfulCreate replicaset/workspace027a7a2667034873-ff6dff45b Created pod: workspace027a7a2667034873-ff6dff45b-h5262
0s Normal Scheduled pod/workspace027a7a2667034873-ff6dff45b-h5262 Successfully assigned divine-chelladurai-che-pmjg9u/workspace027a7a2667034873-ff6dff45b-h5262 to tkc-workers-wnnlk-b55cc6bdd-h75s7
0s Warning FailedAttachVolume pod/workspace027a7a2667034873-ff6dff45b-h5262 Multi-Attach error for volume "pvc-65ab8953-6926-4c53-b0d2-b9ebb2153b74" Volume is already exclusively attached to one node and can't be attached to another
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled down replica set workspace027a7a2667034873-ff6dff45b to 0
0s Normal SuccessfulDelete replicaset/workspace027a7a2667034873-ff6dff45b Deleted pod: workspace027a7a2667034873-ff6dff45b-h5262
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled up replica set workspace027a7a2667034873-ff6dff45b to 1
0s Normal SuccessfulCreate replicaset/workspace027a7a2667034873-ff6dff45b Created pod: workspace027a7a2667034873-ff6dff45b-rzcgr
0s Normal Scheduled pod/workspace027a7a2667034873-ff6dff45b-rzcgr Successfully assigned divine-chelladurai-che-pmjg9u/workspace027a7a2667034873-ff6dff45b-rzcgr to tkc-workers-wnnlk-b55cc6bdd-d6767
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled down replica set workspace027a7a2667034873-ff6dff45b to 0
0s Normal SuccessfulDelete replicaset/workspace027a7a2667034873-ff6dff45b Deleted pod: workspace027a7a2667034873-ff6dff45b-rzcgr
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled up replica set workspace027a7a2667034873-ff6dff45b to 1
0s Normal SuccessfulCreate replicaset/workspace027a7a2667034873-ff6dff45b Created pod: workspace027a7a2667034873-ff6dff45b-fjx7c
0s Normal Scheduled pod/workspace027a7a2667034873-ff6dff45b-fjx7c Successfully assigned divine-chelladurai-che-pmjg9u/workspace027a7a2667034873-ff6dff45b-fjx7c to tkc-workers-wnnlk-b55cc6bdd-d6767
0s Normal SuccessfulAttachVolume pod/workspace027a7a2667034873-ff6dff45b-fjx7c AttachVolume.Attach succeeded for volume "pvc-65ab8953-6926-4c53-b0d2-b9ebb2153b74"
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled down replica set workspace027a7a2667034873-ff6dff45b to 0
0s Normal SuccessfulDelete replicaset/workspace027a7a2667034873-ff6dff45b Deleted pod: workspace027a7a2667034873-ff6dff45b-fjx7c
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled up replica set workspace027a7a2667034873-ff6dff45b to 1
0s Normal SuccessfulCreate replicaset/workspace027a7a2667034873-ff6dff45b Created pod: workspace027a7a2667034873-ff6dff45b-sm2wh
0s Normal Scheduled pod/workspace027a7a2667034873-ff6dff45b-sm2wh Successfully assigned divine-chelladurai-che-pmjg9u/workspace027a7a2667034873-ff6dff45b-sm2wh to tkc-workers-wnnlk-b55cc6bdd-d6767
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled down replica set workspace027a7a2667034873-ff6dff45b to 0
0s Normal SuccessfulDelete replicaset/workspace027a7a2667034873-ff6dff45b Deleted pod: workspace027a7a2667034873-ff6dff45b-sm2wh
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled up replica set workspace027a7a2667034873-ff6dff45b to 1
0s Normal SuccessfulCreate replicaset/workspace027a7a2667034873-ff6dff45b Created pod: workspace027a7a2667034873-ff6dff45b-tjv52
0s Normal Scheduled pod/workspace027a7a2667034873-ff6dff45b-tjv52 Successfully assigned divine-chelladurai-che-pmjg9u/workspace027a7a2667034873-ff6dff45b-tjv52 to tkc-workers-wnnlk-b55cc6bdd-h75s7
0s Normal SuccessfulAttachVolume pod/workspace027a7a2667034873-ff6dff45b-tjv52 AttachVolume.Attach succeeded for volume "pvc-65ab8953-6926-4c53-b0d2-b9ebb2153b74"
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled down replica set workspace027a7a2667034873-ff6dff45b to 0
0s Normal SuccessfulDelete replicaset/workspace027a7a2667034873-ff6dff45b Deleted pod: workspace027a7a2667034873-ff6dff45b-tjv52
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled up replica set workspace027a7a2667034873-ff6dff45b to 1
0s Normal SuccessfulCreate replicaset/workspace027a7a2667034873-ff6dff45b Created pod: workspace027a7a2667034873-ff6dff45b-f5kqn
0s Normal Scheduled pod/workspace027a7a2667034873-ff6dff45b-f5kqn Successfully assigned divine-chelladurai-che-pmjg9u/workspace027a7a2667034873-ff6dff45b-f5kqn to tkc-workers-wnnlk-b55cc6bdd-d6767
0s Warning FailedAttachVolume pod/workspace027a7a2667034873-ff6dff45b-f5kqn Multi-Attach error for volume "pvc-65ab8953-6926-4c53-b0d2-b9ebb2153b74" Volume is already exclusively attached to one node and can't be attached to another
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled down replica set workspace027a7a2667034873-ff6dff45b to 0
0s Normal SuccessfulDelete replicaset/workspace027a7a2667034873-ff6dff45b Deleted pod: workspace027a7a2667034873-ff6dff45b-f5kqn
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled up replica set workspace027a7a2667034873-ff6dff45b to 1
0s Normal Scheduled pod/workspace027a7a2667034873-ff6dff45b-b785j Successfully assigned divine-chelladurai-che-pmjg9u/workspace027a7a2667034873-ff6dff45b-b785j to tkc-workers-wnnlk-b55cc6bdd-d6767
0s Normal SuccessfulCreate replicaset/workspace027a7a2667034873-ff6dff45b Created pod: workspace027a7a2667034873-ff6dff45b-b785j
0s Warning FailedAttachVolume pod/workspace027a7a2667034873-ff6dff45b-b785j Multi-Attach error for volume "pvc-65ab8953-6926-4c53-b0d2-b9ebb2153b74" Volume is already exclusively attached to one node and can't be attached to another
0s Normal SuccessfulDelete replicaset/workspace027a7a2667034873-ff6dff45b Deleted pod: workspace027a7a2667034873-ff6dff45b-b785j
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled down replica set workspace027a7a2667034873-ff6dff45b to 0
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled up replica set workspace027a7a2667034873-ff6dff45b to 1
0s Normal SuccessfulCreate replicaset/workspace027a7a2667034873-ff6dff45b Created pod: workspace027a7a2667034873-ff6dff45b-xs6p5
0s Normal Scheduled pod/workspace027a7a2667034873-ff6dff45b-xs6p5 Successfully assigned divine-chelladurai-che-pmjg9u/workspace027a7a2667034873-ff6dff45b-xs6p5 to tkc-workers-wnnlk-b55cc6bdd-d6767
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled down replica set workspace027a7a2667034873-ff6dff45b to 0
0s Normal SuccessfulDelete replicaset/workspace027a7a2667034873-ff6dff45b (combined from similar events): Deleted pod: workspace027a7a2667034873-ff6dff45b-xs6p5
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled up replica set workspace027a7a2667034873-ff6dff45b to 1
0s Normal SuccessfulCreate replicaset/workspace027a7a2667034873-ff6dff45b (combined from similar events): Created pod: workspace027a7a2667034873-ff6dff45b-r5pq9
0s Normal Scheduled pod/workspace027a7a2667034873-ff6dff45b-r5pq9 Successfully assigned divine-chelladurai-che-pmjg9u/workspace027a7a2667034873-ff6dff45b-r5pq9 to tkc-workers-wnnlk-b55cc6bdd-d6767
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled down replica set workspace027a7a2667034873-ff6dff45b to 0
0s Normal SuccessfulDelete replicaset/workspace027a7a2667034873-ff6dff45b (combined from similar events): Deleted pod: workspace027a7a2667034873-ff6dff45b-r5pq9
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled up replica set workspace027a7a2667034873-ff6dff45b to 1
0s Normal SuccessfulCreate replicaset/workspace027a7a2667034873-ff6dff45b (combined from similar events): Created pod: workspace027a7a2667034873-ff6dff45b-68jjh
0s Normal Scheduled pod/workspace027a7a2667034873-ff6dff45b-68jjh Successfully assigned divine-chelladurai-che-pmjg9u/workspace027a7a2667034873-ff6dff45b-68jjh to tkc-workers-wnnlk-b55cc6bdd-7k767
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled down replica set workspace027a7a2667034873-ff6dff45b to 0
0s Normal SuccessfulDelete replicaset/workspace027a7a2667034873-ff6dff45b (combined from similar events): Deleted pod: workspace027a7a2667034873-ff6dff45b-68jjh
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled up replica set workspace027a7a2667034873-ff6dff45b to 1
0s Normal Scheduled pod/workspace027a7a2667034873-ff6dff45b-pw67p Successfully assigned divine-chelladurai-che-pmjg9u/workspace027a7a2667034873-ff6dff45b-pw67p to tkc-workers-wnnlk-b55cc6bdd-h75s7
0s Normal SuccessfulCreate replicaset/workspace027a7a2667034873-ff6dff45b (combined from similar events): Created pod: workspace027a7a2667034873-ff6dff45b-pw67p
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled down replica set workspace027a7a2667034873-ff6dff45b to 0
0s Normal SuccessfulDelete replicaset/workspace027a7a2667034873-ff6dff45b (combined from similar events): Deleted pod: workspace027a7a2667034873-ff6dff45b-pw67p
0s Normal Scheduled pod/workspace027a7a2667034873-ff6dff45b-z9qpc Successfully assigned divine-chelladurai-che-pmjg9u/workspace027a7a2667034873-ff6dff45b-z9qpc to tkc-workers-wnnlk-b55cc6bdd-d6767
0s Normal SuccessfulCreate replicaset/workspace027a7a2667034873-5486b546c4 Created pod: workspace027a7a2667034873-5486b546c4-hgx89
0s Normal Scheduled pod/workspace027a7a2667034873-5486b546c4-hgx89 Successfully assigned divine-chelladurai-che-pmjg9u/workspace027a7a2667034873-5486b546c4-hgx89 to tkc-workers-wnnlk-b55cc6bdd-d6767
0s Warning FailedMount pod/workspace027a7a2667034873-5486b546c4-hgx89 MountVolume.SetUp failed for volume "automount-configmap-devworkspace-gitconfig" : failed to sync configmap cache: timed out waiting for the condition
0s Warning FailedMount pod/workspace027a7a2667034873-5486b546c4-hgx89 MountVolume.SetUp failed for volume "automount-secret-user-profile" : failed to sync secret cache: timed out waiting for the condition
0s Warning FailedMount pod/workspace027a7a2667034873-5486b546c4-hgx89 MountVolume.SetUp failed for volume "automount-secret-che-server-cert" : failed to sync secret cache: timed out waiting for the condition
0s Warning FailedMount pod/workspace027a7a2667034873-ff6dff45b-lkmmp Unable to attach or mount volumes: unmounted volumes=[automount-secret-che-server-cert automount-secret-user-profile remote-endpoint automount-configmap-devworkspace-gitconfig che-gateway automount-secret-devworkspace-merged-git-credentials automount-configmap-che-trusted-ca-certs claim-devworkspace workspace-metadata kube-api-access-jjnft], unattached volumes=[automount-secret-che-server-cert automount-secret-user-profile remote-endpoint automount-configmap-devworkspace-gitconfig che-gateway automount-secret-devworkspace-merged-git-credentials automount-configmap-che-trusted-ca-certs claim-devworkspace workspace-metadata kube-api-access-jjnft]: timed out waiting for the condition
0s Warning FailedMount pod/workspace027a7a2667034873-ff6dff45b-h5262 Unable to attach or mount volumes: unmounted volumes=[automount-secret-user-profile kube-api-access-wfx5b automount-secret-che-server-cert remote-endpoint automount-secret-devworkspace-merged-git-credentials claim-devworkspace automount-configmap-devworkspace-gitconfig automount-configmap-che-trusted-ca-certs workspace-metadata che-gateway], unattached volumes=[automount-secret-user-profile kube-api-access-wfx5b automount-secret-che-server-cert remote-endpoint automount-secret-devworkspace-merged-git-credentials claim-devworkspace automount-configmap-devworkspace-gitconfig automount-configmap-che-trusted-ca-certs workspace-metadata che-gateway]: timed out waiting for the condition
0s Warning FailedMount pod/workspace027a7a2667034873-ff6dff45b-rzcgr Unable to attach or mount volumes: unmounted volumes=[claim-devworkspace remote-endpoint automount-configmap-devworkspace-gitconfig automount-secret-devworkspace-merged-git-credentials workspace-metadata automount-configmap-che-trusted-ca-certs automount-secret-che-server-cert automount-secret-user-profile kube-api-access-6wgxx che-gateway], unattached volumes=[claim-devworkspace remote-endpoint automount-configmap-devworkspace-gitconfig automount-secret-devworkspace-merged-git-credentials workspace-metadata automount-configmap-che-trusted-ca-certs automount-secret-che-server-cert automount-secret-user-profile kube-api-access-6wgxx che-gateway]: timed out waiting for the condition
0s Warning FailedMount pod/workspace027a7a2667034873-ff6dff45b-fjx7c Unable to attach or mount volumes: unmounted volumes=[che-gateway automount-secret-devworkspace-merged-git-credentials automount-configmap-devworkspace-gitconfig automount-secret-user-profile workspace-metadata claim-devworkspace kube-api-access-fgdrl remote-endpoint automount-configmap-che-trusted-ca-certs automount-secret-che-server-cert], unattached volumes=[che-gateway automount-secret-devworkspace-merged-git-credentials automount-configmap-devworkspace-gitconfig automount-secret-user-profile workspace-metadata claim-devworkspace kube-api-access-fgdrl remote-endpoint automount-configmap-che-trusted-ca-certs automount-secret-che-server-cert]: timed out waiting for the condition
0s Warning FailedMount pod/workspace027a7a2667034873-ff6dff45b-sm2wh Unable to attach or mount volumes: unmounted volumes=[kube-api-access-z64jq automount-secret-che-server-cert automount-secret-user-profile automount-configmap-che-trusted-ca-certs workspace-metadata remote-endpoint automount-secret-devworkspace-merged-git-credentials automount-configmap-devworkspace-gitconfig claim-devworkspace che-gateway], unattached volumes=[kube-api-access-z64jq automount-secret-che-server-cert automount-secret-user-profile automount-configmap-che-trusted-ca-certs workspace-metadata remote-endpoint automount-secret-devworkspace-merged-git-credentials automount-configmap-devworkspace-gitconfig claim-devworkspace che-gateway]: timed out waiting for the condition
0s Warning FailedMount pod/workspace027a7a2667034873-ff6dff45b-tjv52 Unable to attach or mount volumes: unmounted volumes=[automount-secret-user-profile kube-api-access-cw8j9 automount-configmap-devworkspace-gitconfig automount-configmap-che-trusted-ca-certs che-gateway claim-devworkspace automount-secret-devworkspace-merged-git-credentials automount-secret-che-server-cert workspace-metadata remote-endpoint], unattached volumes=[automount-secret-user-profile kube-api-access-cw8j9 automount-configmap-devworkspace-gitconfig automount-configmap-che-trusted-ca-certs che-gateway claim-devworkspace automount-secret-devworkspace-merged-git-credentials automount-secret-che-server-cert workspace-metadata remote-endpoint]: timed out waiting for the condition
0s Warning FailedMount pod/workspace027a7a2667034873-ff6dff45b-f5kqn Unable to attach or mount volumes: unmounted volumes=[kube-api-access-x4zlp automount-secret-che-server-cert workspace-metadata claim-devworkspace automount-secret-devworkspace-merged-git-credentials automount-configmap-che-trusted-ca-certs automount-configmap-devworkspace-gitconfig automount-secret-user-profile che-gateway remote-endpoint], unattached volumes=[kube-api-access-x4zlp automount-secret-che-server-cert workspace-metadata claim-devworkspace automount-secret-devworkspace-merged-git-credentials automount-configmap-che-trusted-ca-certs automount-configmap-devworkspace-gitconfig automount-secret-user-profile che-gateway remote-endpoint]: timed out waiting for the condition
0s Warning FailedMount pod/workspace027a7a2667034873-ff6dff45b-b785j Unable to attach or mount volumes: unmounted volumes=[workspace-metadata che-gateway remote-endpoint automount-configmap-che-trusted-ca-certs automount-secret-user-profile claim-devworkspace automount-secret-devworkspace-merged-git-credentials automount-secret-che-server-cert automount-configmap-devworkspace-gitconfig kube-api-access-xq7bb], unattached volumes=[workspace-metadata che-gateway remote-endpoint automount-configmap-che-trusted-ca-certs automount-secret-user-profile claim-devworkspace automount-secret-devworkspace-merged-git-credentials automount-secret-che-server-cert automount-configmap-devworkspace-gitconfig kube-api-access-xq7bb]: timed out waiting for the condition
0s Warning FailedAttachVolume pod/workspace027a7a2667034873-ff6dff45b-b785j AttachVolume.Attach failed for volume "pvc-65ab8953-6926-4c53-b0d2-b9ebb2153b74" : Attach timeout for volume df63f994-eaeb-47a8-ac23-0e222cac3b10-65ab8953-6926-4c53-b0d2-b9ebb2153b74
0s Warning FailedMount pod/workspace027a7a2667034873-ff6dff45b-xs6p5 Unable to attach or mount volumes: unmounted volumes=[automount-configmap-che-trusted-ca-certs kube-api-access-hcc7h remote-endpoint automount-configmap-devworkspace-gitconfig automount-secret-che-server-cert che-gateway claim-devworkspace automount-secret-devworkspace-merged-git-credentials automount-secret-user-profile workspace-metadata], unattached volumes=[automount-configmap-che-trusted-ca-certs kube-api-access-hcc7h remote-endpoint automount-configmap-devworkspace-gitconfig automount-secret-che-server-cert che-gateway claim-devworkspace automount-secret-devworkspace-merged-git-credentials automount-secret-user-profile workspace-metadata]: timed out waiting for the condition
0s Warning FailedMount pod/workspace027a7a2667034873-ff6dff45b-r5pq9 Unable to attach or mount volumes: unmounted volumes=[remote-endpoint automount-secret-che-server-cert kube-api-access-h7lsg automount-configmap-che-trusted-ca-certs automount-secret-user-profile che-gateway claim-devworkspace automount-configmap-devworkspace-gitconfig automount-secret-devworkspace-merged-git-credentials workspace-metadata], unattached volumes=[remote-endpoint automount-secret-che-server-cert kube-api-access-h7lsg automount-configmap-che-trusted-ca-certs automount-secret-user-profile che-gateway claim-devworkspace automount-configmap-devworkspace-gitconfig automount-secret-devworkspace-merged-git-credentials workspace-metadata]: timed out waiting for the condition
0s Warning FailedMount pod/workspace027a7a2667034873-ff6dff45b-68jjh Unable to attach or mount volumes: unmounted volumes=[che-gateway remote-endpoint automount-secret-user-profile workspace-metadata kube-api-access-ns9d8 automount-secret-devworkspace-merged-git-credentials automount-configmap-che-trusted-ca-certs claim-devworkspace automount-configmap-devworkspace-gitconfig automount-secret-che-server-cert], unattached volumes=[che-gateway remote-endpoint automount-secret-user-profile workspace-metadata kube-api-access-ns9d8 automount-secret-devworkspace-merged-git-credentials automount-configmap-che-trusted-ca-certs claim-devworkspace automount-configmap-devworkspace-gitconfig automount-secret-che-server-cert]: timed out waiting for the condition
0s Warning FailedMount pod/workspace027a7a2667034873-ff6dff45b-pw67p Unable to attach or mount volumes: unmounted volumes=[automount-configmap-che-trusted-ca-certs automount-secret-che-server-cert che-gateway remote-endpoint automount-configmap-devworkspace-gitconfig claim-devworkspace automount-secret-user-profile workspace-metadata automount-secret-devworkspace-merged-git-credentials kube-api-access-84lsm], unattached volumes=[automount-configmap-che-trusted-ca-certs automount-secret-che-server-cert che-gateway remote-endpoint automount-configmap-devworkspace-gitconfig claim-devworkspace automount-secret-user-profile workspace-metadata automount-secret-devworkspace-merged-git-credentials kube-api-access-84lsm]: timed out waiting for the condition
0s Warning FailedMount pod/workspace027a7a2667034873-ff6dff45b-z9qpc Unable to attach or mount volumes: unmounted volumes=[automount-secret-devworkspace-merged-git-credentials remote-endpoint automount-secret-user-profile claim-devworkspace kube-api-access-cs2wm che-gateway automount-configmap-devworkspace-gitconfig automount-configmap-che-trusted-ca-certs automount-secret-che-server-cert workspace-metadata], unattached volumes=[automount-secret-devworkspace-merged-git-credentials remote-endpoint automount-secret-user-profile claim-devworkspace kube-api-access-cs2wm che-gateway automount-configmap-devworkspace-gitconfig automount-configmap-che-trusted-ca-certs automount-secret-che-server-cert workspace-metadata]: timed out waiting for the condition
0s Warning FailedAttachVolume pod/workspace027a7a2667034873-5486b546c4-hgx89 AttachVolume.Attach failed for volume "pvc-65ab8953-6926-4c53-b0d2-b9ebb2153b74" : rpc error: code = Aborted desc = pending
0s Warning FailedAttachVolume pod/workspace027a7a2667034873-5486b546c4-hgx89 AttachVolume.Attach failed for volume "pvc-65ab8953-6926-4c53-b0d2-b9ebb2153b74" : rpc error: code = Aborted desc = pending
0s Warning FailedAttachVolume pod/workspace027a7a2667034873-5486b546c4-hgx89 AttachVolume.Attach failed for volume "pvc-65ab8953-6926-4c53-b0d2-b9ebb2153b74" : rpc error: code = Aborted desc = pending
0s Warning FailedAttachVolume pod/workspace027a7a2667034873-5486b546c4-hgx89 AttachVolume.Attach failed for volume "pvc-65ab8953-6926-4c53-b0d2-b9ebb2153b74" : rpc error: code = Aborted desc = pending
0s Warning FailedMount pod/workspace027a7a2667034873-5486b546c4-hgx89 Unable to attach or mount volumes: unmounted volumes=[claim-devworkspace], unattached volumes=[kube-api-access-ckwzb workspace-metadata che-gateway automount-secret-devworkspace-merged-git-credentials remote-endpoint automount-configmap-che-trusted-ca-certs automount-configmap-devworkspace-gitconfig automount-secret-user-profile automount-secret-che-server-cert claim-devworkspace]: timed out waiting for the condition
0s Normal SuccessfulDelete replicaset/workspace027a7a2667034873-5486b546c4 Deleted pod: workspace027a7a2667034873-5486b546c4-hgx89
0s Normal SuccessfulCreate replicaset/workspace027a7a2667034873-76f86b7c54 Created pod: workspace027a7a2667034873-76f86b7c54-6f2hb
0s Normal Scheduled pod/workspace027a7a2667034873-76f86b7c54-6f2hb Successfully assigned divine-chelladurai-che-pmjg9u/workspace027a7a2667034873-76f86b7c54-6f2hb to tkc-workers-wnnlk-b55cc6bdd-d6767
0s Normal SuccessfulDelete replicaset/workspace027a7a2667034873-76f86b7c54 Deleted pod: workspace027a7a2667034873-76f86b7c54-6f2hb
0s Warning FailedAttachVolume pod/workspace027a7a2667034873-76f86b7c54-6f2hb AttachVolume.Attach failed for volume "pvc-65ab8953-6926-4c53-b0d2-b9ebb2153b74" : rpc error: code = Aborted desc = pending
0s Normal SuccessfulCreate replicaset/workspace027a7a2667034873-76f86b7c54 Created pod: workspace027a7a2667034873-76f86b7c54-cgkw4
0s Normal Scheduled pod/workspace027a7a2667034873-76f86b7c54-cgkw4 Successfully assigned divine-chelladurai-che-pmjg9u/workspace027a7a2667034873-76f86b7c54-cgkw4 to tkc-workers-wnnlk-b55cc6bdd-h75s7
0s Normal SuccessfulDelete replicaset/workspace027a7a2667034873-76f86b7c54 Deleted pod: workspace027a7a2667034873-76f86b7c54-cgkw4
0s Normal SuccessfulCreate replicaset/workspace027a7a2667034873-5486b546c4 Created pod: workspace027a7a2667034873-5486b546c4-hq9cb
0s Normal Scheduled pod/workspace027a7a2667034873-5486b546c4-hq9cb Successfully assigned divine-chelladurai-che-pmjg9u/workspace027a7a2667034873-5486b546c4-hq9cb to tkc-workers-wnnlk-b55cc6bdd-d6767
0s Warning FailedAttachVolume pod/workspace027a7a2667034873-5486b546c4-hq9cb AttachVolume.Attach failed for volume "pvc-65ab8953-6926-4c53-b0d2-b9ebb2153b74" : rpc error: code = Aborted desc = pending
0s Normal SuccessfulAttachVolume pod/workspace027a7a2667034873-5486b546c4-hq9cb AttachVolume.Attach succeeded for volume "pvc-65ab8953-6926-4c53-b0d2-b9ebb2153b74"
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled down replica set workspace027a7a2667034873-5486b546c4 to 0
0s Normal SuccessfulDelete replicaset/workspace027a7a2667034873-5486b546c4 Deleted pod: workspace027a7a2667034873-5486b546c4-hq9cb
0s Warning FailedMount pod/workspace027a7a2667034873-76f86b7c54-6f2hb Unable to attach or mount volumes: unmounted volumes=[che-gateway automount-secret-che-server-cert workspace-metadata kube-api-access-jn2dh automount-secret-user-profile remote-endpoint claim-devworkspace automount-secret-devworkspace-merged-git-credentials automount-configmap-devworkspace-gitconfig automount-configmap-che-trusted-ca-certs], unattached volumes=[che-gateway automount-secret-che-server-cert workspace-metadata kube-api-access-jn2dh automount-secret-user-profile remote-endpoint claim-devworkspace automount-secret-devworkspace-merged-git-credentials automount-configmap-devworkspace-gitconfig automount-configmap-che-trusted-ca-certs]: timed out waiting for the condition
0s Warning FailedMount pod/workspace027a7a2667034873-76f86b7c54-cgkw4 Unable to attach or mount volumes: unmounted volumes=[automount-secret-user-profile che-gateway claim-devworkspace remote-endpoint automount-secret-devworkspace-merged-git-credentials automount-configmap-devworkspace-gitconfig automount-configmap-che-trusted-ca-certs workspace-metadata automount-secret-che-server-cert kube-api-access-thcmd], unattached volumes=[automount-secret-user-profile che-gateway claim-devworkspace remote-endpoint automount-secret-devworkspace-merged-git-credentials automount-configmap-devworkspace-gitconfig automount-configmap-che-trusted-ca-certs workspace-metadata automount-secret-che-server-cert kube-api-access-thcmd]: timed out waiting for the condition
0s Warning FailedMount pod/workspace027a7a2667034873-5486b546c4-hq9cb Unable to attach or mount volumes: unmounted volumes=[automount-configmap-devworkspace-gitconfig workspace-metadata claim-devworkspace automount-secret-devworkspace-merged-git-credentials automount-secret-che-server-cert che-gateway remote-endpoint kube-api-access-92s9t automount-configmap-che-trusted-ca-certs automount-secret-user-profile], unattached volumes=[automount-configmap-devworkspace-gitconfig workspace-metadata claim-devworkspace automount-secret-devworkspace-merged-git-credentials automount-secret-che-server-cert che-gateway remote-endpoint kube-api-access-92s9t automount-configmap-che-trusted-ca-certs automount-secret-user-profile]: timed out waiting for the condition
i clicked restart button from the screenshot mentioned in my above comment.. i received below error again
0s Normal SuccessfulCreate replicaset/workspace027a7a2667034873-5486b546c4 Created pod: workspace027a7a2667034873-5486b546c4-gwdzp
0s Normal Scheduled pod/workspace027a7a2667034873-5486b546c4-gwdzp Successfully assigned divine-chelladurai-che-pmjg9u/workspace027a7a2667034873-5486b546c4-gwdzp to tkc-workers-wnnlk-b55cc6bdd-d6767
0s Normal SuccessfulAttachVolume pod/workspace027a7a2667034873-5486b546c4-gwdzp AttachVolume.Attach succeeded for volume "pvc-65ab8953-6926-4c53-b0d2-b9ebb2153b74"
0s Normal SuccessfulDelete replicaset/workspace027a7a2667034873-5486b546c4 Deleted pod: workspace027a7a2667034873-5486b546c4-gwdzp
0s Normal SuccessfulCreate replicaset/workspace027a7a2667034873-5486b546c4 Created pod: workspace027a7a2667034873-5486b546c4-sh7jw
0s Normal Scheduled pod/workspace027a7a2667034873-5486b546c4-sh7jw Successfully assigned divine-chelladurai-che-pmjg9u/workspace027a7a2667034873-5486b546c4-sh7jw to tkc-workers-wnnlk-b55cc6bdd-h75s7
0s Warning FailedAttachVolume pod/workspace027a7a2667034873-5486b546c4-sh7jw Multi-Attach error for volume "pvc-65ab8953-6926-4c53-b0d2-b9ebb2153b74" Volume is already exclusively attached to one node and can't be attached to another
0s Normal SuccessfulDelete replicaset/workspace027a7a2667034873-5486b546c4 Deleted pod: workspace027a7a2667034873-5486b546c4-sh7jw
0s Warning FailedMount pod/workspace027a7a2667034873-5486b546c4-gwdzp Unable to attach or mount volumes: unmounted volumes=[automount-secret-che-server-cert che-gateway remote-endpoint automount-configmap-devworkspace-gitconfig kube-api-access-pntz6 automount-secret-devworkspace-merged-git-credentials automount-secret-user-profile workspace-metadata claim-devworkspace automount-configmap-che-trusted-ca-certs], unattached volumes=[automount-secret-che-server-cert che-gateway remote-endpoint automount-configmap-devworkspace-gitconfig kube-api-access-pntz6 automount-secret-devworkspace-merged-git-credentials automount-secret-user-profile workspace-metadata claim-devworkspace automount-configmap-che-trusted-ca-certs]: timed out waiting for the condition
0s Normal SuccessfulCreate replicaset/workspace027a7a2667034873-ff6dff45b (combined from similar events): Created pod: workspace027a7a2667034873-ff6dff45b-gckqv
0s Normal Scheduled pod/workspace027a7a2667034873-ff6dff45b-gckqv Successfully assigned divine-chelladurai-che-pmjg9u/workspace027a7a2667034873-ff6dff45b-gckqv to tkc-workers-wnnlk-b55cc6bdd-d6767
0s Warning FailedMount pod/workspace027a7a2667034873-5486b546c4-sh7jw Unable to attach or mount volumes: unmounted volumes=[automount-secret-devworkspace-merged-git-credentials automount-configmap-che-trusted-ca-certs automount-secret-user-profile automount-secret-che-server-cert che-gateway claim-devworkspace remote-endpoint automount-configmap-devworkspace-gitconfig workspace-metadata kube-api-access-fl2rr], unattached volumes=[automount-secret-devworkspace-merged-git-credentials automount-configmap-che-trusted-ca-certs automount-secret-user-profile automount-secret-che-server-cert che-gateway claim-devworkspace remote-endpoint automount-configmap-devworkspace-gitconfig workspace-metadata kube-api-access-fl2rr]: timed out waiting for the condition
0s Warning FailedAttachVolume pod/workspace027a7a2667034873-ff6dff45b-gckqv AttachVolume.Attach failed for volume "pvc-65ab8953-6926-4c53-b0d2-b9ebb2153b74" : Attach timeout for volume df63f994-eaeb-47a8-ac23-0e222cac3b10-65ab8953-6926-4c53-b0d2-b9ebb2153b74
0s Warning FailedMount pod/workspace027a7a2667034873-ff6dff45b-gckqv Unable to attach or mount volumes: unmounted volumes=[claim-devworkspace], unattached volumes=[automount-secret-devworkspace-merged-git-credentials workspace-metadata claim-devworkspace remote-endpoint automount-secret-user-profile automount-configmap-devworkspace-gitconfig automount-configmap-che-trusted-ca-certs che-gateway automount-secret-che-server-cert kube-api-access-k4fzp]: timed out waiting for the condition
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled down replica set workspace027a7a2667034873-ff6dff45b to 0
0s Normal SuccessfulDelete replicaset/workspace027a7a2667034873-ff6dff45b Deleted pod: workspace027a7a2667034873-ff6dff45b-gckqv
0s Warning FailedAttachVolume pod/workspace027a7a2667034873-ff6dff45b-gckqv AttachVolume.Attach failed for volume "pvc-65ab8953-6926-4c53-b0d2-b9ebb2153b74" : Attach timeout for volume df63f994-eaeb-47a8-ac23-0e222cac3b10-65ab8953-6926-4c53-b0d2-b9ebb2153b74
0s Warning FailedMount pod/workspace027a7a2667034873-ff6dff45b-gckqv Unable to attach or mount volumes: unmounted volumes=[automount-secret-devworkspace-merged-git-credentials automount-configmap-devworkspace-gitconfig automount-secret-che-server-cert workspace-metadata claim-devworkspace che-gateway automount-secret-user-profile kube-api-access-k4fzp automount-configmap-che-trusted-ca-certs remote-endpoint], unattached volumes=[automount-secret-devworkspace-merged-git-credentials automount-configmap-devworkspace-gitconfig automount-secret-che-server-cert workspace-metadata claim-devworkspace che-gateway automount-secret-user-profile kube-api-access-k4fzp automount-configmap-che-trusted-ca-certs remote-endpoint]: timed out waiting for the condition
why does the log show AttachVolume.Attach succeeded for volume "pvc-65ab8953-6926-4c53-b0d2-b9ebb2153b74"
then after few seconds it shows AttachVolume.Attach failed for volume "pvc-65ab8953-6926-4c53-b0d2-b9ebb2153b74 ?
please help me to fix this issue
It is interesting to check logs from:
che-serverche-operatordevworkspace-controller- storage provisioner
logs of workspace pod failing couple of times (reloaded the page) and then succeeding at the end (after page reload)
dchelladura-a01:vtapbotservice divine$ kubectl get events -n divine-chelladurai-che-pmjg9u -w
LAST SEEN TYPE REASON OBJECT MESSAGE
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled up replica set workspace027a7a2667034873-76f86b7c54 to 1
0s Normal SuccessfulCreate replicaset/workspace027a7a2667034873-76f86b7c54 Created pod: workspace027a7a2667034873-76f86b7c54-jx2lx
0s Normal Scheduled pod/workspace027a7a2667034873-76f86b7c54-jx2lx Successfully assigned divine-chelladurai-che-pmjg9u/workspace027a7a2667034873-76f86b7c54-jx2lx to tkc-workers-wnnlk-b55cc6bdd-d6767
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled down replica set workspace027a7a2667034873-76f86b7c54 to 0
0s Normal SuccessfulDelete replicaset/workspace027a7a2667034873-76f86b7c54 Deleted pod: workspace027a7a2667034873-76f86b7c54-jx2lx
0s Normal SuccessfulAttachVolume pod/workspace027a7a2667034873-76f86b7c54-jx2lx AttachVolume.Attach succeeded for volume "pvc-65ab8953-6926-4c53-b0d2-b9ebb2153b74"
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled up replica set workspace027a7a2667034873-76f86b7c54 to 1
0s Normal SuccessfulCreate replicaset/workspace027a7a2667034873-76f86b7c54 Created pod: workspace027a7a2667034873-76f86b7c54-lrcr2
0s Normal Scheduled pod/workspace027a7a2667034873-76f86b7c54-lrcr2 Successfully assigned divine-chelladurai-che-pmjg9u/workspace027a7a2667034873-76f86b7c54-lrcr2 to tkc-workers-wnnlk-b55cc6bdd-h75s7
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled down replica set workspace027a7a2667034873-76f86b7c54 to 0
0s Normal SuccessfulDelete replicaset/workspace027a7a2667034873-76f86b7c54 Deleted pod: workspace027a7a2667034873-76f86b7c54-lrcr2
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled up replica set workspace027a7a2667034873-ff6dff45b to 1
0s Normal SuccessfulCreate replicaset/workspace027a7a2667034873-ff6dff45b Created pod: workspace027a7a2667034873-ff6dff45b-tn5rm
0s Normal Scheduled pod/workspace027a7a2667034873-ff6dff45b-tn5rm Successfully assigned divine-chelladurai-che-pmjg9u/workspace027a7a2667034873-ff6dff45b-tn5rm to tkc-workers-wnnlk-b55cc6bdd-h75s7
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled down replica set workspace027a7a2667034873-ff6dff45b to 0
0s Normal SuccessfulDelete replicaset/workspace027a7a2667034873-ff6dff45b Deleted pod: workspace027a7a2667034873-ff6dff45b-tn5rm
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled up replica set workspace027a7a2667034873-ff6dff45b to 1
0s Normal SuccessfulCreate replicaset/workspace027a7a2667034873-ff6dff45b Created pod: workspace027a7a2667034873-ff6dff45b-7klww
0s Normal Scheduled pod/workspace027a7a2667034873-ff6dff45b-7klww Successfully assigned divine-chelladurai-che-pmjg9u/workspace027a7a2667034873-ff6dff45b-7klww to tkc-workers-wnnlk-b55cc6bdd-d6767
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled down replica set workspace027a7a2667034873-ff6dff45b to 0
0s Normal SuccessfulDelete replicaset/workspace027a7a2667034873-ff6dff45b Deleted pod: workspace027a7a2667034873-ff6dff45b-7klww
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled up replica set workspace027a7a2667034873-ff6dff45b to 1
0s Normal SuccessfulCreate replicaset/workspace027a7a2667034873-ff6dff45b Created pod: workspace027a7a2667034873-ff6dff45b-n6t6s
0s Normal Scheduled pod/workspace027a7a2667034873-ff6dff45b-n6t6s Successfully assigned divine-chelladurai-che-pmjg9u/workspace027a7a2667034873-ff6dff45b-n6t6s to tkc-workers-wnnlk-b55cc6bdd-h75s7
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled down replica set workspace027a7a2667034873-ff6dff45b to 0
0s Normal SuccessfulDelete replicaset/workspace027a7a2667034873-ff6dff45b Deleted pod: workspace027a7a2667034873-ff6dff45b-n6t6s
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled up replica set workspace027a7a2667034873-5486b546c4 to 1
0s Normal SuccessfulCreate replicaset/workspace027a7a2667034873-5486b546c4 Created pod: workspace027a7a2667034873-5486b546c4-xs7h8
0s Normal Scheduled pod/workspace027a7a2667034873-5486b546c4-xs7h8 Successfully assigned divine-chelladurai-che-pmjg9u/workspace027a7a2667034873-5486b546c4-xs7h8 to tkc-workers-wnnlk-b55cc6bdd-h75s7
0s Warning FailedMount pod/workspace027a7a2667034873-76f86b7c54-jx2lx Unable to attach or mount volumes: unmounted volumes=[automount-configmap-che-trusted-ca-certs automount-secret-user-profile remote-endpoint claim-devworkspace kube-api-access-ll7b7 workspace-metadata che-gateway automount-configmap-devworkspace-gitconfig automount-secret-devworkspace-merged-git-credentials automount-secret-che-server-cert], unattached volumes=[automount-configmap-che-trusted-ca-certs automount-secret-user-profile remote-endpoint claim-devworkspace kube-api-access-ll7b7 workspace-metadata che-gateway automount-configmap-devworkspace-gitconfig automount-secret-devworkspace-merged-git-credentials automount-secret-che-server-cert]: timed out waiting for the condition
0s Warning FailedAttachVolume pod/workspace027a7a2667034873-76f86b7c54-lrcr2 AttachVolume.Attach failed for volume "pvc-65ab8953-6926-4c53-b0d2-b9ebb2153b74" : Attach timeout for volume df63f994-eaeb-47a8-ac23-0e222cac3b10-65ab8953-6926-4c53-b0d2-b9ebb2153b74
0s Warning FailedMount pod/workspace027a7a2667034873-76f86b7c54-lrcr2 Unable to attach or mount volumes: unmounted volumes=[che-gateway workspace-metadata automount-configmap-che-trusted-ca-certs automount-secret-user-profile remote-endpoint kube-api-access-4gfls automount-secret-devworkspace-merged-git-credentials automount-secret-che-server-cert claim-devworkspace automount-configmap-devworkspace-gitconfig], unattached volumes=[che-gateway workspace-metadata automount-configmap-che-trusted-ca-certs automount-secret-user-profile remote-endpoint kube-api-access-4gfls automount-secret-devworkspace-merged-git-credentials automount-secret-che-server-cert claim-devworkspace automount-configmap-devworkspace-gitconfig]: timed out waiting for the condition
0s Warning FailedMount pod/workspace027a7a2667034873-ff6dff45b-tn5rm Unable to attach or mount volumes: unmounted volumes=[automount-configmap-devworkspace-gitconfig automount-configmap-che-trusted-ca-certs automount-secret-user-profile automount-secret-devworkspace-merged-git-credentials remote-endpoint workspace-metadata kube-api-access-2bzqs automount-secret-che-server-cert che-gateway claim-devworkspace], unattached volumes=[automount-configmap-devworkspace-gitconfig automount-configmap-che-trusted-ca-certs automount-secret-user-profile automount-secret-devworkspace-merged-git-credentials remote-endpoint workspace-metadata kube-api-access-2bzqs automount-secret-che-server-cert che-gateway claim-devworkspace]: timed out waiting for the condition
0s Warning FailedMount pod/workspace027a7a2667034873-ff6dff45b-7klww Unable to attach or mount volumes: unmounted volumes=[kube-api-access-jzgqg workspace-metadata claim-devworkspace automount-secret-devworkspace-merged-git-credentials automount-secret-user-profile automount-secret-che-server-cert remote-endpoint automount-configmap-devworkspace-gitconfig automount-configmap-che-trusted-ca-certs che-gateway], unattached volumes=[kube-api-access-jzgqg workspace-metadata claim-devworkspace automount-secret-devworkspace-merged-git-credentials automount-secret-user-profile automount-secret-che-server-cert remote-endpoint automount-configmap-devworkspace-gitconfig automount-configmap-che-trusted-ca-certs che-gateway]: timed out waiting for the condition
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled down replica set workspace027a7a2667034873-5486b546c4 to 0
0s Normal SuccessfulDelete replicaset/workspace027a7a2667034873-5486b546c4 Deleted pod: workspace027a7a2667034873-5486b546c4-xs7h8
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled up replica set workspace027a7a2667034873-ff6dff45b to 1
0s Normal SuccessfulCreate replicaset/workspace027a7a2667034873-ff6dff45b Created pod: workspace027a7a2667034873-ff6dff45b-n8pzs
0s Normal Scheduled pod/workspace027a7a2667034873-ff6dff45b-n8pzs Successfully assigned divine-chelladurai-che-pmjg9u/workspace027a7a2667034873-ff6dff45b-n8pzs to tkc-workers-wnnlk-b55cc6bdd-d6767
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled down replica set workspace027a7a2667034873-ff6dff45b to 0
0s Normal SuccessfulDelete replicaset/workspace027a7a2667034873-ff6dff45b Deleted pod: workspace027a7a2667034873-ff6dff45b-n8pzs
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled up replica set workspace027a7a2667034873-ff6dff45b to 1
0s Normal SuccessfulCreate replicaset/workspace027a7a2667034873-ff6dff45b Created pod: workspace027a7a2667034873-ff6dff45b-s9f29
0s Normal Scheduled pod/workspace027a7a2667034873-ff6dff45b-s9f29 Successfully assigned divine-chelladurai-che-pmjg9u/workspace027a7a2667034873-ff6dff45b-s9f29 to tkc-workers-wnnlk-b55cc6bdd-h75s7
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled down replica set workspace027a7a2667034873-ff6dff45b to 0
0s Normal SuccessfulDelete replicaset/workspace027a7a2667034873-ff6dff45b Deleted pod: workspace027a7a2667034873-ff6dff45b-s9f29
1s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled up replica set workspace027a7a2667034873-ff6dff45b to 1
0s Normal SuccessfulCreate replicaset/workspace027a7a2667034873-ff6dff45b Created pod: workspace027a7a2667034873-ff6dff45b-cmf2f
0s Normal Scheduled pod/workspace027a7a2667034873-ff6dff45b-cmf2f Successfully assigned divine-chelladurai-che-pmjg9u/workspace027a7a2667034873-ff6dff45b-cmf2f to tkc-workers-wnnlk-b55cc6bdd-d6767
0s Warning FailedMount pod/workspace027a7a2667034873-ff6dff45b-n6t6s Unable to attach or mount volumes: unmounted volumes=[automount-secret-che-server-cert workspace-metadata claim-devworkspace automount-configmap-devworkspace-gitconfig che-gateway automount-secret-devworkspace-merged-git-credentials automount-secret-user-profile remote-endpoint automount-configmap-che-trusted-ca-certs kube-api-access-r4w74], unattached volumes=[automount-secret-che-server-cert workspace-metadata claim-devworkspace automount-configmap-devworkspace-gitconfig che-gateway automount-secret-devworkspace-merged-git-credentials automount-secret-user-profile remote-endpoint automount-configmap-che-trusted-ca-certs kube-api-access-r4w74]: timed out waiting for the condition
0s Warning FailedAttachVolume pod/workspace027a7a2667034873-5486b546c4-xs7h8 AttachVolume.Attach failed for volume "pvc-65ab8953-6926-4c53-b0d2-b9ebb2153b74" : rpc error: code = Aborted desc = pending
0s Warning FailedMount pod/workspace027a7a2667034873-5486b546c4-xs7h8 Unable to attach or mount volumes: unmounted volumes=[automount-secret-che-server-cert workspace-metadata kube-api-access-kstsr claim-devworkspace automount-configmap-che-trusted-ca-certs remote-endpoint automount-secret-devworkspace-merged-git-credentials automount-configmap-devworkspace-gitconfig che-gateway automount-secret-user-profile], unattached volumes=[automount-secret-che-server-cert workspace-metadata kube-api-access-kstsr claim-devworkspace automount-configmap-che-trusted-ca-certs remote-endpoint automount-secret-devworkspace-merged-git-credentials automount-configmap-devworkspace-gitconfig che-gateway automount-secret-user-profile]: timed out waiting for the condition
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled down replica set workspace027a7a2667034873-ff6dff45b to 0
0s Normal SuccessfulDelete replicaset/workspace027a7a2667034873-ff6dff45b Deleted pod: workspace027a7a2667034873-ff6dff45b-cmf2f
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled up replica set workspace027a7a2667034873-ff6dff45b to 1
0s Normal SuccessfulCreate replicaset/workspace027a7a2667034873-ff6dff45b Created pod: workspace027a7a2667034873-ff6dff45b-pwcgh
0s Normal Scheduled pod/workspace027a7a2667034873-ff6dff45b-pwcgh Successfully assigned divine-chelladurai-che-pmjg9u/workspace027a7a2667034873-ff6dff45b-pwcgh to tkc-workers-wnnlk-b55cc6bdd-h75s7
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled down replica set workspace027a7a2667034873-ff6dff45b to 0
0s Normal SuccessfulDelete replicaset/workspace027a7a2667034873-ff6dff45b Deleted pod: workspace027a7a2667034873-ff6dff45b-pwcgh
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled up replica set workspace027a7a2667034873-ff6dff45b to 1
0s Normal SuccessfulCreate replicaset/workspace027a7a2667034873-ff6dff45b Created pod: workspace027a7a2667034873-ff6dff45b-zzqx9
0s Normal Scheduled pod/workspace027a7a2667034873-ff6dff45b-zzqx9 Successfully assigned divine-chelladurai-che-pmjg9u/workspace027a7a2667034873-ff6dff45b-zzqx9 to tkc-workers-wnnlk-b55cc6bdd-h75s7
0s Normal SuccessfulAttachVolume pod/workspace027a7a2667034873-ff6dff45b-cmf2f AttachVolume.Attach succeeded for volume "pvc-65ab8953-6926-4c53-b0d2-b9ebb2153b74"
0s Warning FailedAttachVolume pod/workspace027a7a2667034873-ff6dff45b-zzqx9 Multi-Attach error for volume "pvc-65ab8953-6926-4c53-b0d2-b9ebb2153b74" Volume is already exclusively attached to one node and can't be attached to another
0s Normal SuccessfulAttachVolume pod/workspace027a7a2667034873-ff6dff45b-zzqx9 AttachVolume.Attach succeeded for volume "pvc-65ab8953-6926-4c53-b0d2-b9ebb2153b74"
0s Warning FailedMount pod/workspace027a7a2667034873-ff6dff45b-n8pzs Unable to attach or mount volumes: unmounted volumes=[che-gateway workspace-metadata automount-secret-devworkspace-merged-git-credentials automount-secret-user-profile remote-endpoint automount-secret-che-server-cert kube-api-access-mqhz9 claim-devworkspace automount-configmap-devworkspace-gitconfig automount-configmap-che-trusted-ca-certs], unattached volumes=[che-gateway workspace-metadata automount-secret-devworkspace-merged-git-credentials automount-secret-user-profile remote-endpoint automount-secret-che-server-cert kube-api-access-mqhz9 claim-devworkspace automount-configmap-devworkspace-gitconfig automount-configmap-che-trusted-ca-certs]: timed out waiting for the condition
0s Warning FailedMount pod/workspace027a7a2667034873-ff6dff45b-s9f29 Unable to attach or mount volumes: unmounted volumes=[workspace-metadata automount-configmap-che-trusted-ca-certs automount-secret-user-profile claim-devworkspace remote-endpoint automount-secret-devworkspace-merged-git-credentials automount-configmap-devworkspace-gitconfig automount-secret-che-server-cert che-gateway kube-api-access-w6p2c], unattached volumes=[workspace-metadata automount-configmap-che-trusted-ca-certs automount-secret-user-profile claim-devworkspace remote-endpoint automount-secret-devworkspace-merged-git-credentials automount-configmap-devworkspace-gitconfig automount-secret-che-server-cert che-gateway kube-api-access-w6p2c]: timed out waiting for the condition
0s Warning FailedMount pod/workspace027a7a2667034873-ff6dff45b-cmf2f Unable to attach or mount volumes: unmounted volumes=[automount-configmap-che-trusted-ca-certs automount-secret-che-server-cert claim-devworkspace automount-secret-user-profile automount-configmap-devworkspace-gitconfig workspace-metadata kube-api-access-26tgm remote-endpoint automount-secret-devworkspace-merged-git-credentials che-gateway], unattached volumes=[automount-configmap-che-trusted-ca-certs automount-secret-che-server-cert claim-devworkspace automount-secret-user-profile automount-configmap-devworkspace-gitconfig workspace-metadata kube-api-access-26tgm remote-endpoint automount-secret-devworkspace-merged-git-credentials che-gateway]: timed out waiting for the condition
0s Normal Pulling pod/workspace027a7a2667034873-ff6dff45b-zzqx9 Pulling image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next"
0s Normal Pulled pod/workspace027a7a2667034873-ff6dff45b-zzqx9 Successfully pulled image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next" in 375.105785ms
0s Normal Created pod/workspace027a7a2667034873-ff6dff45b-zzqx9 Created container remote-runtime-injector
0s Normal Started pod/workspace027a7a2667034873-ff6dff45b-zzqx9 Started container remote-runtime-injector
0s Normal Pulling pod/workspace027a7a2667034873-ff6dff45b-zzqx9 Pulling image "quay.io/devfile/project-clone:v0.15.2"
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled down replica set workspace027a7a2667034873-ff6dff45b to 0
0s Normal SuccessfulDelete replicaset/workspace027a7a2667034873-ff6dff45b Deleted pod: workspace027a7a2667034873-ff6dff45b-zzqx9
0s Normal Pulled pod/workspace027a7a2667034873-ff6dff45b-zzqx9 Successfully pulled image "quay.io/devfile/project-clone:v0.15.2" in 354.502969ms
0s Normal Created pod/workspace027a7a2667034873-ff6dff45b-zzqx9 Created container project-clone
0s Normal Started pod/workspace027a7a2667034873-ff6dff45b-zzqx9 Started container project-clone
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled up replica set workspace027a7a2667034873-ff6dff45b to 1
0s Normal SuccessfulCreate replicaset/workspace027a7a2667034873-ff6dff45b Created pod: workspace027a7a2667034873-ff6dff45b-vx2l5
0s Normal Scheduled pod/workspace027a7a2667034873-ff6dff45b-vx2l5 Successfully assigned divine-chelladurai-che-pmjg9u/workspace027a7a2667034873-ff6dff45b-vx2l5 to tkc-workers-wnnlk-b55cc6bdd-d6767
0s Warning FailedAttachVolume pod/workspace027a7a2667034873-ff6dff45b-vx2l5 Multi-Attach error for volume "pvc-65ab8953-6926-4c53-b0d2-b9ebb2153b74" Volume is already exclusively attached to one node and can't be attached to another
0s Normal Pulling pod/workspace027a7a2667034873-ff6dff45b-zzqx9 Pulling image "quay.io/eclipse/che-theia:next"
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled down replica set workspace027a7a2667034873-ff6dff45b to 0
0s Normal SuccessfulDelete replicaset/workspace027a7a2667034873-ff6dff45b Deleted pod: workspace027a7a2667034873-ff6dff45b-vx2l5
0s Normal Pulled pod/workspace027a7a2667034873-ff6dff45b-zzqx9 Successfully pulled image "quay.io/eclipse/che-theia:next" in 349.038665ms
0s Normal Created pod/workspace027a7a2667034873-ff6dff45b-zzqx9 Created container theia-ide
0s Warning Failed pod/workspace027a7a2667034873-ff6dff45b-zzqx9 Error: failed to create containerd task: OCI runtime create failed: container_linux.go:364: creating new parent process caused: container_linux.go:2005: running lstat on namespace path "/proc/1204/ns/ipc" caused: lstat /proc/1204/ns/ipc: no such file or directory: unknown
0s Normal Pulling pod/workspace027a7a2667034873-ff6dff45b-zzqx9 Pulling image "quay.io/eclipse/che-machine-exec:next"
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled up replica set workspace027a7a2667034873-ff6dff45b to 1
0s Normal Scheduled pod/workspace027a7a2667034873-ff6dff45b-vgrqx Successfully assigned divine-chelladurai-che-pmjg9u/workspace027a7a2667034873-ff6dff45b-vgrqx to tkc-workers-wnnlk-b55cc6bdd-d6767
0s Normal SuccessfulCreate replicaset/workspace027a7a2667034873-ff6dff45b (combined from similar events): Created pod: workspace027a7a2667034873-ff6dff45b-vgrqx
0s Warning FailedAttachVolume pod/workspace027a7a2667034873-ff6dff45b-vgrqx Multi-Attach error for volume "pvc-65ab8953-6926-4c53-b0d2-b9ebb2153b74" Volume is already exclusively attached to one node and can't be attached to another
0s Normal Pulled pod/workspace027a7a2667034873-ff6dff45b-zzqx9 Successfully pulled image "quay.io/eclipse/che-machine-exec:next" in 359.303474ms
0s Warning Failed pod/workspace027a7a2667034873-ff6dff45b-zzqx9 Error: failed to get sandbox container task: no running task found: task d2dc652019636557b08c2b53958e0c25a99d7746ed701d641e70bd0bab5b385b not found: not found
0s Normal Pulling pod/workspace027a7a2667034873-ff6dff45b-zzqx9 Pulling image "artfact-prd.vmware.com:5001/qedocker/eclipseche/customopenjdk8:v10"
0s Warning FailedMount pod/workspace027a7a2667034873-ff6dff45b-vx2l5 MountVolume.SetUp failed for volume "automount-secret-che-server-cert" : failed to sync secret cache: timed out waiting for the condition
0s Warning FailedMount pod/workspace027a7a2667034873-ff6dff45b-vx2l5 MountVolume.SetUp failed for volume "automount-configmap-che-trusted-ca-certs" : failed to sync configmap cache: timed out waiting for the condition
0s Warning FailedMount pod/workspace027a7a2667034873-ff6dff45b-vx2l5 MountVolume.SetUp failed for volume "automount-configmap-devworkspace-gitconfig" : failed to sync configmap cache: timed out waiting for the condition
0s Warning FailedMount pod/workspace027a7a2667034873-ff6dff45b-vx2l5 MountVolume.SetUp failed for volume "automount-secret-user-profile" : failed to sync secret cache: timed out waiting for the condition
0s Warning FailedMount pod/workspace027a7a2667034873-ff6dff45b-vx2l5 MountVolume.SetUp failed for volume "automount-secret-devworkspace-merged-git-credentials" : failed to sync secret cache: timed out waiting for the condition
1s Normal Pulled pod/workspace027a7a2667034873-ff6dff45b-zzqx9 Successfully pulled image "artfact-prd.vmware.com:5001/qedocker/eclipseche/customopenjdk8:v10" in 541.414697ms
1s Warning Failed pod/workspace027a7a2667034873-ff6dff45b-zzqx9 Error: cannot find volume "claim-devworkspace" to mount into container "javacontainer"
1s Normal Pulled pod/workspace027a7a2667034873-ff6dff45b-zzqx9 Container image "quay.io/eclipse/che--traefik:v2.8.1-4e52a5e2495484f5e19a49edfd2f652b0bce7b3603fa0df545ed90168ffae1c3" already present on machine
1s Warning Failed pod/workspace027a7a2667034873-ff6dff45b-zzqx9 Error: cannot find volume "che-gateway" to mount into container "che-gateway"
0s Warning FailedMount pod/workspace027a7a2667034873-ff6dff45b-vx2l5 MountVolume.SetUp failed for volume "workspace-metadata" : failed to sync configmap cache: timed out waiting for the condition
0s Warning FailedMount pod/workspace027a7a2667034873-ff6dff45b-vgrqx MountVolume.SetUp failed for volume "automount-secret-che-server-cert" : failed to sync secret cache: timed out waiting for the condition
0s Warning FailedMount pod/workspace027a7a2667034873-ff6dff45b-vgrqx MountVolume.SetUp failed for volume "automount-configmap-devworkspace-gitconfig" : failed to sync configmap cache: timed out waiting for the condition
0s Warning FailedMount pod/workspace027a7a2667034873-ff6dff45b-vgrqx MountVolume.SetUp failed for volume "automount-secret-user-profile" : failed to sync secret cache: timed out waiting for the condition
0s Warning FailedMount pod/workspace027a7a2667034873-ff6dff45b-vgrqx MountVolume.SetUp failed for volume "automount-secret-devworkspace-merged-git-credentials" : failed to sync secret cache: timed out waiting for the condition
0s Warning FailedMount pod/workspace027a7a2667034873-ff6dff45b-vx2l5 MountVolume.SetUp failed for volume "automount-secret-devworkspace-merged-git-credentials" : failed to sync secret cache: timed out waiting for the condition
0s Warning FailedMount pod/workspace027a7a2667034873-ff6dff45b-vx2l5 MountVolume.SetUp failed for volume "automount-configmap-devworkspace-gitconfig" : failed to sync configmap cache: timed out waiting for the condition
1s Warning FailedMount pod/workspace027a7a2667034873-ff6dff45b-vx2l5 MountVolume.SetUp failed for volume "automount-secret-user-profile" : failed to sync secret cache: timed out waiting for the condition
0s Normal SuccessfulAttachVolume pod/workspace027a7a2667034873-ff6dff45b-vgrqx AttachVolume.Attach succeeded for volume "pvc-65ab8953-6926-4c53-b0d2-b9ebb2153b74"
0s Normal SuccessfulDelete replicaset/workspace027a7a2667034873-ff6dff45b (combined from similar events): Deleted pod: workspace027a7a2667034873-ff6dff45b-vgrqx
0s Normal ScalingReplicaSet deployment/workspace027a7a2667034873 Scaled up replica set workspace027a7a2667034873-ff6dff45b to 1
0s Normal SuccessfulCreate replicaset/workspace027a7a2667034873-ff6dff45b (combined from similar events): Created pod: workspace027a7a2667034873-ff6dff45b-jssvp
0s Normal Scheduled pod/workspace027a7a2667034873-ff6dff45b-jssvp Successfully assigned divine-chelladurai-che-pmjg9u/workspace027a7a2667034873-ff6dff45b-jssvp to tkc-workers-wnnlk-b55cc6bdd-d6767
1s Normal SuccessfulDelete replicaset/workspace027a7a2667034873-ff6dff45b (combined from similar events): Deleted pod: workspace027a7a2667034873-ff6dff45b-jssvp
0s Normal SuccessfulCreate replicaset/workspace027a7a2667034873-ff6dff45b (combined from similar events): Created pod: workspace027a7a2667034873-ff6dff45b-4g64l
0s Normal Scheduled pod/workspace027a7a2667034873-ff6dff45b-4g64l Successfully assigned divine-chelladurai-che-pmjg9u/workspace027a7a2667034873-ff6dff45b-4g64l to tkc-workers-wnnlk-b55cc6bdd-h75s7
0s Normal SuccessfulDelete replicaset/workspace027a7a2667034873-ff6dff45b (combined from similar events): Deleted pod: workspace027a7a2667034873-ff6dff45b-4g64l
0s Normal SuccessfulCreate replicaset/workspace027a7a2667034873-ff6dff45b (combined from similar events): Created pod: workspace027a7a2667034873-ff6dff45b-9smgc
0s Normal Scheduled pod/workspace027a7a2667034873-ff6dff45b-9smgc Successfully assigned divine-chelladurai-che-pmjg9u/workspace027a7a2667034873-ff6dff45b-9smgc to tkc-workers-wnnlk-b55cc6bdd-d6767
0s Warning FailedMount pod/workspace027a7a2667034873-ff6dff45b-pwcgh Unable to attach or mount volumes: unmounted volumes=[remote-endpoint che-gateway automount-configmap-devworkspace-gitconfig automount-secret-user-profile workspace-metadata automount-configmap-che-trusted-ca-certs automount-secret-che-server-cert kube-api-access-b6mxk claim-devworkspace automount-secret-devworkspace-merged-git-credentials], unattached volumes=[remote-endpoint che-gateway automount-configmap-devworkspace-gitconfig automount-secret-user-profile workspace-metadata automount-configmap-che-trusted-ca-certs automount-secret-che-server-cert kube-api-access-b6mxk claim-devworkspace automount-secret-devworkspace-merged-git-credentials]: timed out waiting for the condition
0s Normal SuccessfulDelete replicaset/workspace027a7a2667034873-ff6dff45b (combined from similar events): Deleted pod: workspace027a7a2667034873-ff6dff45b-9smgc
0s Normal Scheduled pod/workspace027a7a2667034873-5486b546c4-s2mh9 Successfully assigned divine-chelladurai-che-pmjg9u/workspace027a7a2667034873-5486b546c4-s2mh9 to tkc-workers-wnnlk-b55cc6bdd-d6767
0s Normal SuccessfulCreate replicaset/workspace027a7a2667034873-5486b546c4 Created pod: workspace027a7a2667034873-5486b546c4-s2mh9
0s Normal SuccessfulDelete replicaset/workspace027a7a2667034873-5486b546c4 Deleted pod: workspace027a7a2667034873-5486b546c4-s2mh9
0s Normal Scheduled pod/workspace027a7a2667034873-ff6dff45b-wcqjh Successfully assigned divine-chelladurai-che-pmjg9u/workspace027a7a2667034873-ff6dff45b-wcqjh to tkc-workers-wnnlk-b55cc6bdd-d6767
0s Warning FailedMount pod/workspace027a7a2667034873-ff6dff45b-vx2l5 Unable to attach or mount volumes: unmounted volumes=[automount-secret-user-profile kube-api-access-9bctp che-gateway automount-secret-devworkspace-merged-git-credentials remote-endpoint automount-configmap-che-trusted-ca-certs automount-configmap-devworkspace-gitconfig automount-secret-che-server-cert workspace-metadata claim-devworkspace], unattached volumes=[automount-secret-user-profile kube-api-access-9bctp che-gateway automount-secret-devworkspace-merged-git-credentials remote-endpoint automount-configmap-che-trusted-ca-certs automount-configmap-devworkspace-gitconfig automount-secret-che-server-cert workspace-metadata claim-devworkspace]: timed out waiting for the condition
0s Warning FailedMount pod/workspace027a7a2667034873-ff6dff45b-vgrqx Unable to attach or mount volumes: unmounted volumes=[claim-devworkspace automount-secret-devworkspace-merged-git-credentials automount-secret-user-profile che-gateway remote-endpoint automount-configmap-devworkspace-gitconfig kube-api-access-kw4hm automount-configmap-che-trusted-ca-certs automount-secret-che-server-cert workspace-metadata], unattached volumes=[claim-devworkspace automount-secret-devworkspace-merged-git-credentials automount-secret-user-profile che-gateway remote-endpoint automount-configmap-devworkspace-gitconfig kube-api-access-kw4hm automount-configmap-che-trusted-ca-certs automount-secret-che-server-cert workspace-metadata]: timed out waiting for the condition
0s Warning FailedAttachVolume pod/workspace027a7a2667034873-ff6dff45b-jssvp AttachVolume.Attach failed for volume "pvc-65ab8953-6926-4c53-b0d2-b9ebb2153b74" : Attach timeout for volume df63f994-eaeb-47a8-ac23-0e222cac3b10-65ab8953-6926-4c53-b0d2-b9ebb2153b74
0s Warning FailedMount pod/workspace027a7a2667034873-ff6dff45b-jssvp Unable to attach or mount volumes: unmounted volumes=[automount-secret-devworkspace-merged-git-credentials automount-secret-che-server-cert che-gateway workspace-metadata claim-devworkspace automount-secret-user-profile remote-endpoint automount-configmap-devworkspace-gitconfig kube-api-access-dklsn automount-configmap-che-trusted-ca-certs], unattached volumes=[automount-secret-devworkspace-merged-git-credentials automount-secret-che-server-cert che-gateway workspace-metadata claim-devworkspace automount-secret-user-profile remote-endpoint automount-configmap-devworkspace-gitconfig kube-api-access-dklsn automount-configmap-che-trusted-ca-certs]: timed out waiting for the condition
0s Warning FailedMount pod/workspace027a7a2667034873-ff6dff45b-4g64l Unable to attach or mount volumes: unmounted volumes=[automount-secret-devworkspace-merged-git-credentials automount-configmap-devworkspace-gitconfig kube-api-access-swr22 che-gateway claim-devworkspace automount-configmap-che-trusted-ca-certs workspace-metadata remote-endpoint automount-secret-che-server-cert automount-secret-user-profile], unattached volumes=[automount-secret-devworkspace-merged-git-credentials automount-configmap-devworkspace-gitconfig kube-api-access-swr22 che-gateway claim-devworkspace automount-configmap-che-trusted-ca-certs workspace-metadata remote-endpoint automount-secret-che-server-cert automount-secret-user-profile]: timed out waiting for the condition
0s Warning FailedMount pod/workspace027a7a2667034873-ff6dff45b-9smgc Unable to attach or mount volumes: unmounted volumes=[che-gateway kube-api-access-r8sb2 remote-endpoint automount-secret-devworkspace-merged-git-credentials automount-secret-che-server-cert automount-configmap-devworkspace-gitconfig automount-configmap-che-trusted-ca-certs automount-secret-user-profile workspace-metadata claim-devworkspace], unattached volumes=[che-gateway kube-api-access-r8sb2 remote-endpoint automount-secret-devworkspace-merged-git-credentials automount-secret-che-server-cert automount-configmap-devworkspace-gitconfig automount-configmap-che-trusted-ca-certs automount-secret-user-profile workspace-metadata claim-devworkspace]: timed out waiting for the condition
0s Normal Scheduled pod/workspace027a7a2667034873-ff6dff45b-xs49g Successfully assigned divine-chelladurai-che-pmjg9u/workspace027a7a2667034873-ff6dff45b-xs49g to tkc-workers-wnnlk-b55cc6bdd-h75s7
0s Warning FailedMount pod/workspace027a7a2667034873-5486b546c4-s2mh9 Unable to attach or mount volumes: unmounted volumes=[remote-endpoint automount-configmap-che-trusted-ca-certs kube-api-access-rzdqx workspace-metadata che-gateway automount-secret-devworkspace-merged-git-credentials automount-configmap-devworkspace-gitconfig automount-secret-che-server-cert automount-secret-user-profile claim-devworkspace], unattached volumes=[remote-endpoint automount-configmap-che-trusted-ca-certs kube-api-access-rzdqx workspace-metadata che-gateway automount-secret-devworkspace-merged-git-credentials automount-configmap-devworkspace-gitconfig automount-secret-che-server-cert automount-secret-user-profile claim-devworkspace]: timed out waiting for the condition
0s Warning FailedMount pod/workspace027a7a2667034873-ff6dff45b-wcqjh Unable to attach or mount volumes: unmounted volumes=[kube-api-access-8hwgj workspace-metadata remote-endpoint automount-secret-devworkspace-merged-git-credentials automount-configmap-che-trusted-ca-certs automount-configmap-devworkspace-gitconfig automount-secret-che-server-cert claim-devworkspace che-gateway automount-secret-user-profile], unattached volumes=[kube-api-access-8hwgj workspace-metadata remote-endpoint automount-secret-devworkspace-merged-git-credentials automount-configmap-che-trusted-ca-certs automount-configmap-devworkspace-gitconfig automount-secret-che-server-cert claim-devworkspace che-gateway automount-secret-user-profile]: timed out waiting for the condition
0s Normal Scheduled pod/workspace027a7a2667034873-ff6dff45b-srdbr Successfully assigned divine-chelladurai-che-pmjg9u/workspace027a7a2667034873-ff6dff45b-srdbr to tkc-workers-wnnlk-b55cc6bdd-d6767
0s Normal Scheduled pod/workspace027a7a2667034873-ff6dff45b-vchld Successfully assigned divine-chelladurai-che-pmjg9u/workspace027a7a2667034873-ff6dff45b-vchld to tkc-workers-wnnlk-b55cc6bdd-h75s7
0s Warning FailedAttachVolume pod/workspace027a7a2667034873-ff6dff45b-wcqjh AttachVolume.Attach failed for volume "pvc-65ab8953-6926-4c53-b0d2-b9ebb2153b74" : Attach timeout for volume df63f994-eaeb-47a8-ac23-0e222cac3b10-65ab8953-6926-4c53-b0d2-b9ebb2153b74
0s Normal SuccessfulAttachVolume pod/workspace027a7a2667034873-ff6dff45b-vchld AttachVolume.Attach succeeded for volume "pvc-65ab8953-6926-4c53-b0d2-b9ebb2153b74"
0s Normal Pulling pod/workspace027a7a2667034873-ff6dff45b-vchld Pulling image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next"
0s Normal Pulled pod/workspace027a7a2667034873-ff6dff45b-vchld Successfully pulled image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next" in 376.936287ms
0s Normal Created pod/workspace027a7a2667034873-ff6dff45b-vchld Created container remote-runtime-injector
0s Normal Started pod/workspace027a7a2667034873-ff6dff45b-vchld Started container remote-runtime-injector
0s Normal Pulling pod/workspace027a7a2667034873-ff6dff45b-vchld Pulling image "quay.io/devfile/project-clone:v0.15.2"
0s Normal Pulled pod/workspace027a7a2667034873-ff6dff45b-vchld Successfully pulled image "quay.io/devfile/project-clone:v0.15.2" in 338.868359ms
0s Normal Created pod/workspace027a7a2667034873-ff6dff45b-vchld Created container project-clone
0s Normal Started pod/workspace027a7a2667034873-ff6dff45b-vchld Started container project-clone
0s Normal SuccessfulCreate replicaset/workspace027a7a2667034873-5486b546c4 Created pod: workspace027a7a2667034873-5486b546c4-qhh9f
0s Normal Scheduled pod/workspace027a7a2667034873-5486b546c4-qhh9f Successfully assigned divine-chelladurai-che-pmjg9u/workspace027a7a2667034873-5486b546c4-qhh9f to tkc-workers-wnnlk-b55cc6bdd-h75s7
0s Normal Pulling pod/workspace027a7a2667034873-5486b546c4-qhh9f Pulling image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next"
0s Normal Pulled pod/workspace027a7a2667034873-5486b546c4-qhh9f Successfully pulled image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next" in 342.615395ms
0s Normal Created pod/workspace027a7a2667034873-5486b546c4-qhh9f Created container remote-runtime-injector
0s Normal Started pod/workspace027a7a2667034873-5486b546c4-qhh9f Started container remote-runtime-injector
0s Normal Pulling pod/workspace027a7a2667034873-5486b546c4-qhh9f Pulling image "quay.io/devfile/project-clone:v0.15.2"
0s Normal Pulled pod/workspace027a7a2667034873-5486b546c4-qhh9f Successfully pulled image "quay.io/devfile/project-clone:v0.15.2" in 363.5747ms
0s Normal Created pod/workspace027a7a2667034873-5486b546c4-qhh9f Created container project-clone
0s Normal Started pod/workspace027a7a2667034873-5486b546c4-qhh9f Started container project-clone
0s Normal Pulling pod/workspace027a7a2667034873-5486b546c4-qhh9f Pulling image "quay.io/eclipse/che-theia:next"
0s Normal Pulled pod/workspace027a7a2667034873-5486b546c4-qhh9f Successfully pulled image "quay.io/eclipse/che-theia:next" in 348.331125ms
0s Normal Created pod/workspace027a7a2667034873-5486b546c4-qhh9f Created container theia-ide
0s Normal Started pod/workspace027a7a2667034873-5486b546c4-qhh9f Started container theia-ide
0s Normal Pulling pod/workspace027a7a2667034873-5486b546c4-qhh9f Pulling image "quay.io/eclipse/che-machine-exec:next"
0s Normal Pulled pod/workspace027a7a2667034873-5486b546c4-qhh9f Successfully pulled image "quay.io/eclipse/che-machine-exec:next" in 337.755862ms
0s Normal Created pod/workspace027a7a2667034873-5486b546c4-qhh9f Created container che-machine-exec
0s Normal Started pod/workspace027a7a2667034873-5486b546c4-qhh9f Started container che-machine-exec
0s Normal Pulling pod/workspace027a7a2667034873-5486b546c4-qhh9f Pulling image "artfact-prd.vmware.com:5001/qedocker/eclipseche/customopenjdk8:v10"
0s Normal Pulled pod/workspace027a7a2667034873-5486b546c4-qhh9f Successfully pulled image "artfact-prd.vmware.com:5001/qedocker/eclipseche/customopenjdk8:v10" in 528.25324ms
0s Normal Created pod/workspace027a7a2667034873-5486b546c4-qhh9f Created container javacontainer
0s Normal Started pod/workspace027a7a2667034873-5486b546c4-qhh9f Started container javacontainer
0s Normal Pulled pod/workspace027a7a2667034873-5486b546c4-qhh9f Container image "quay.io/eclipse/che--traefik:v2.8.1-4e52a5e2495484f5e19a49edfd2f652b0bce7b3603fa0df545ed90168ffae1c3" already present on machine
0s Normal Created pod/workspace027a7a2667034873-5486b546c4-qhh9f Created container che-gateway
0s Normal Started pod/workspace027a7a2667034873-5486b546c4-qhh9f Started container che-gateway
0s Warning FailedMount pod/workspace027a7a2667034873-ff6dff45b-xs49g Unable to attach or mount volumes: unmounted volumes=[automount-secret-che-server-cert automount-secret-devworkspace-merged-git-credentials automount-secret-user-profile workspace-metadata kube-api-access-znp7z automount-configmap-devworkspace-gitconfig che-gateway claim-devworkspace remote-endpoint automount-configmap-che-trusted-ca-certs], unattached volumes=[automount-secret-che-server-cert automount-secret-devworkspace-merged-git-credentials automount-secret-user-profile workspace-metadata kube-api-access-znp7z automount-configmap-devworkspace-gitconfig che-gateway claim-devworkspace remote-endpoint automount-configmap-che-trusted-ca-certs]: timed out waiting for the condition
0s Warning FailedMount pod/workspace027a7a2667034873-ff6dff45b-srdbr Unable to attach or mount volumes: unmounted volumes=[automount-secret-devworkspace-merged-git-credentials automount-configmap-che-trusted-ca-certs workspace-metadata remote-endpoint automount-configmap-devworkspace-gitconfig kube-api-access-2hb77 che-gateway claim-devworkspace automount-secret-che-server-cert automount-secret-user-profile], unattached volumes=[automount-secret-devworkspace-merged-git-credentials automount-configmap-che-trusted-ca-certs workspace-metadata remote-endpoint automount-configmap-devworkspace-gitconfig kube-api-access-2hb77 che-gateway claim-devworkspace automount-secret-che-server-cert automount-secret-user-profile]: timed out waiting for the condition
i will observe the logs in the pods you have mentioned and monitor it
@tolusha
but , i did some analysis and this is my observation of my current issue
Assume i have 1 master node and 3 workernodes (workernode1, workernode2, workernode3 ) in my k8s cluster
-
my
podAgets created onworkernode1, andpodArequestsPVCtostorageclassand received aPVwithReadWriteOnceasaccessModes. this is successful. thePVCstate is bounded -
then after an hour, my
podAgets deleted and recreated , this time after recreation ,podAgets created onworkernode2.PVCstate did not change,PVCstate is stillboundedbut thePVhasReadWriteOnceasaccessModes, so even ifpodAis created onworkernode2,podAis unable to claim the volume. i feel this is the reason for my issue
do you think my observation is valid or incorrect? please share your thoughts on this @tolusha
i'm asking the above question because i received this error in the logs posted in my above comments log 0s Warning FailedAttachVolume pod/workspace7722130dbd9049ee-5f5fbbc678-lnqbh Multi-Attach error for volume "pvc-80278a68-7f9a-4e24-83b5-ff4fdf993a35" Volume is already exclusively attached to one node and can't be attached to another
should i change the accessModes to ReadWriteMany from ReadWriteOnce in my PV to fix this issue?
below is the PVC , PV, Storage class configuration in my eclipse-che setup
Persistent volume claim
apiVersion: v1
items:
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
pv.kubernetes.io/bind-completed: "yes"
pv.kubernetes.io/bound-by-controller: "yes"
volume.beta.kubernetes.io/storage-provisioner: csi.vsphere.vmware.com
volumehealth.storage.kubernetes.io/health: accessible
volumehealth.storage.kubernetes.io/health-timestamp: Fri Aug 5 18:42:22 UTC
2022
creationTimestamp: "2022-08-05T18:40:57Z"
finalizers:
- kubernetes.io/pvc-protection
name: claim-devworkspace
namespace: divine-chelladurai-che-pmjg9u
resourceVersion: "22924989"
uid: 65ab8953-6926-4c53-b0d2-b9ebb2153b74
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
storageClassName: vmc-workload-storage-policy-cluster-1
volumeMode: Filesystem
volumeName: pvc-65ab8953-6926-4c53-b0d2-b9ebb2153b74
status:
accessModes:
- ReadWriteOnce
capacity:
storage: 10Gi
phase: Bound
kind: List
metadata:
resourceVersion: ""
selfLink: ""
storage class
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
annotations:
storageclass.kubernetes.io/is-default-class: "true"
creationTimestamp: "2022-06-15T05:57:58Z"
labels:
isSyncedFromSupervisor: "yes"
name: vmc-workload-storage-policy-cluster-1
resourceVersion: "161"
uid: 3e85fd47-ad27-4c61-bca5-44ef2b8410c7
parameters:
svStorageClass: vmc-workload-storage-policy-cluster-1
provisioner: csi.vsphere.vmware.com
reclaimPolicy: Delete
volumeBindingMode: Immediate
persistent volume
apiVersion: v1
kind: PersistentVolume
metadata:
annotations:
pv.kubernetes.io/provisioned-by: csi.vsphere.vmware.com
creationTimestamp: "2022-08-05T18:40:59Z"
finalizers:
- kubernetes.io/pv-protection
- external-attacher/csi-vsphere-vmware-com
name: pvc-65ab8953-6926-4c53-b0d2-b9ebb2153b74
resourceVersion: "23093922"
uid: f22d2721-0e6a-49b9-a45b-0992a25f2626
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 10Gi
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: claim-devworkspace
namespace: divine-chelladurai-che-pmjg9u
resourceVersion: "22924484"
uid: 65ab8953-6926-4c53-b0d2-b9ebb2153b74
csi:
driver: csi.vsphere.vmware.com
fsType: ext4
volumeAttributes:
storage.kubernetes.io/csiProvisionerIdentity: 1655847079993-8081-csi.vsphere.vmware.com
type: vSphere CNS Block Volume
volumeHandle: df63f994-eaeb-47a8-ac23-0e222cac3b10-65ab8953-6926-4c53-b0d2-b9ebb2153b74
persistentVolumeReclaimPolicy: Delete
storageClassName: vmc-workload-storage-policy-cluster-1
volumeMode: Filesystem
status:
phase: Bound
@tolusha
i was able to analyze and gather more information on the issue.
i have 1 master node and 3 worker nodes in my k8s cluster (workernode1, workernode2, workernode3).
when the workspace pod (workspace5ea824993e324b81-766fcf945c-xdd2f) is started in workernode1, the pvc is created and pv is allocated and attached to workernode1,and PV/PVC is bounded to the workspace pod in workernode1. PVC is created and PV with accessMode: ReadWriteOnce is allocated to the workspace pod.
When the che-theia IDE is shutdown, the workspace pod is deleted. Now when the workspace is opened again, the workspace pod might get created at workernode2 or workernode3 or workernode1. Lets Assume that my workspace pod was attempted to be created in workernode2. In this case, the PV which was previously created is attached to workernode1, this has to be detached and attached to workernode2, so that this PV can be claimed by PVC and bound to the pod. This process long time to provision and it fails most of the time. Rarely this gets passed.
@tolusha please share your thoughts on this.
please help me to fix this issue. i can share you any details that are needed for analysis. i have been stuck with this issue too long.
Persistent volume claim
apiVersion: v1
items:
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
pv.kubernetes.io/bind-completed: "yes"
pv.kubernetes.io/bound-by-controller: "yes"
volume.beta.kubernetes.io/storage-provisioner: csi.vsphere.vmware.com
volumehealth.storage.kubernetes.io/health: accessible
volumehealth.storage.kubernetes.io/health-timestamp: Fri Aug 5 18:42:22 UTC
2022
creationTimestamp: "2022-08-05T18:40:57Z"
finalizers:
- kubernetes.io/pvc-protection
name: claim-devworkspace
namespace: divine-chelladurai-che-pmjg9u
resourceVersion: "22924989"
uid: 65ab8953-6926-4c53-b0d2-b9ebb2153b74
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
storageClassName: vmc-workload-storage-policy-cluster-1
volumeMode: Filesystem
volumeName: pvc-65ab8953-6926-4c53-b0d2-b9ebb2153b74
status:
accessModes:
- ReadWriteOnce
capacity:
storage: 10Gi
phase: Bound
kind: List
metadata:
resourceVersion: ""
selfLink: ""
storage class
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
annotations:
storageclass.kubernetes.io/is-default-class: "true"
creationTimestamp: "2022-06-15T05:57:58Z"
labels:
isSyncedFromSupervisor: "yes"
name: vmc-workload-storage-policy-cluster-1
resourceVersion: "161"
uid: 3e85fd47-ad27-4c61-bca5-44ef2b8410c7
parameters:
svStorageClass: vmc-workload-storage-policy-cluster-1
provisioner: csi.vsphere.vmware.com
reclaimPolicy: Delete
volumeBindingMode: Immediate
persistent volume
apiVersion: v1
kind: PersistentVolume
metadata:
annotations:
pv.kubernetes.io/provisioned-by: csi.vsphere.vmware.com
creationTimestamp: "2022-08-05T18:40:59Z"
finalizers:
- kubernetes.io/pv-protection
- external-attacher/csi-vsphere-vmware-com
name: pvc-65ab8953-6926-4c53-b0d2-b9ebb2153b74
resourceVersion: "23093922"
uid: f22d2721-0e6a-49b9-a45b-0992a25f2626
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 10Gi
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: claim-devworkspace
namespace: divine-chelladurai-che-pmjg9u
resourceVersion: "22924484"
uid: 65ab8953-6926-4c53-b0d2-b9ebb2153b74
csi:
driver: csi.vsphere.vmware.com
fsType: ext4
volumeAttributes:
storage.kubernetes.io/csiProvisionerIdentity: 1655847079993-8081-csi.vsphere.vmware.com
type: vSphere CNS Block Volume
volumeHandle: df63f994-eaeb-47a8-ac23-0e222cac3b10-65ab8953-6926-4c53-b0d2-b9ebb2153b74
persistentVolumeReclaimPolicy: Delete
storageClassName: vmc-workload-storage-policy-cluster-1
volumeMode: Filesystem
status:
phase: Bound
@tolusha i'm stuck with this issue now. please share your feedback on this.
@tolusha i have used nodeSelector, i can see my PV and workspace container running in same workerNode even after deletion and re-creation of workspace. but i still face the same issue. i will retest it and share you the feedback.
@tolusha
i have used nodeSelector. Now both my workspace and pv are always getting created in same workernode even after mutliple restarts. But still i'm facing this issue.
i'm receiving below error while starting workspace, any idea on this? is my workspace not starting because of this?.
After several retries to start the workspace, i received below logs in project-clone container, now the workspace started.
Please let me know what is the reason for this issue and how to fix this issue
2022/08/18 12:35:58 Using temporary directory /projects/project-clone-3127608582
2022/08/18 12:35:58 Read DevWorkspace at /devworkspace-metadata/flattened.devworkspace.yaml
2022/08/18 12:35:58 Processing project cbfsel-project
2022/08/18 12:35:58 Cloning project cbfsel-project to /projects/project-clone-3127608582/cbfsel-project
Cloning into '/projects/project-clone-3127608582/cbfsel-project'...
fatal: unable to get credential storage lock in 1000 ms: Permission denied
Updating files: 41% (1748/4190)
Updating files: 42% (1760/4190)
Updating files: 43% (1802/4190)
Updating files: 44% (1844/4190)
Updating files: 45% (1886/4190)
Updating files: 46% (1928/4190)
Updating files: 47% (1970/4190)
Updating files: 48% (2012/4190)
Updating files: 49% (2054/4190)
Updating files: 50% (2095/4190)
Updating files: 51% (2137/4190)
Updating files: 52% (2179/4190)
Updating files: 53% (2221/4190)
Updating files: 54% (2263/4190)
Updating files: 55% (2305/4190)
Updating files: 56% (2347/4190)
Updating files: 57% (2389/4190)
Updating files: 58% (2431/4190)
Updating files: 59% (2473/4190)
Updating files: 60% (2514/4190)
Updating files: 61% (2556/4190)
Updating files: 62% (2598/4190)
Updating files: 63% (2640/4190)
Updating files: 64% (2682/4190)
Updating files: 65% (2724/4190)
Updating files: 66% (2766/4190)
Updating files: 67% (2808/4190)
Updating files: 68% (2850/4190)
Updating files: 69% (2892/4190)
Updating files: 70% (2933/4190)
Updating files: 71% (2975/4190)
Updating files: 72% (3017/4190)
Updating files: 73% (3059/4190)
Updating files: 74% (3101/4190)
Updating files: 75% (3143/4190)
Updating files: 76% (3185/4190)
Updating files: 77% (3227/4190)
Updating files: 78% (3269/4190)
Updating files: 79% (3311/4190)
Updating files: 80% (3352/4190)
Updating files: 81% (3394/4190)
Updating files: 82% (3436/4190)
Updating files: 83% (3478/4190)
Updating files: 84% (3520/4190)
Updating files: 85% (3562/4190)
Updating files: 86% (3604/4190)
Updating files: 87% (3646/4190)
Updating files: 88% (3688/4190)
Updating files: 89% (3730/4190)
Updating files: 90% (3771/4190)
Updating files: 91% (3813/4190)
Updating files: 92% (3855/4190)
Updating files: 93% (3897/4190)
Updating files: 94% (3939/4190)
Updating files: 95% (3981/4190)
Updating files: 96% (4023/4190)
Updating files: 96% (4032/4190)
Updating files: 97% (4065/4190)
Updating files: 98% (4107/4190)
Updating files: 99% (4149/4190)
Updating files: 100% (4190/4190)
Updating files: 100% (4190/4190), done.
2022/08/18 12:37:40 Cloned project cbfsel-project to /projects/project-clone-3127608582/cbfsel-project
2022/08/18 12:37:40 Setting up remotes for project cbfsel-project
fatal: unable to get credential storage lock in 1000 ms: Permission denied
2022/08/18 12:37:40 Fetched remote origin at https://gitlab.eng.vmware.com/dchelladurai/cbf-sel.git
2022/08/18 12:37:41 Encountered error while setting up project cbfsel-project: failed to checkout revision: failed to read remote origin: authentication required
Logs from 8/18/2022, 6:05:58 PM
below is devworkspace-controller logs
{"level":"info","ts":1660829259.0051994,"logger":"controllers.DevWorkspace","msg":"Waiting on deployment to be ready","Request.Namespace":"d-sindhuja-che-a6q94i","Request.Name":"cbfsel-repo","devworkspace_id":"workspaceeb49234a892b48eb"}
{"level":"info","ts":1660829264.0147173,"logger":"controllers.DevWorkspace","msg":"Reconciling Workspace","Request.Namespace":"d-sindhuja-che-a6q94i","Request.Name":"cbfsel-repo","devworkspace_id":"workspaceeb49234a892b48eb"}
{"level":"info","ts":1660829264.0569215,"logger":"controllers.DevWorkspace","msg":"Waiting on deployment to be ready","Request.Namespace":"d-sindhuja-che-a6q94i","Request.Name":"cbfsel-repo","devworkspace_id":"workspaceeb49234a892b48eb"}
{"level":"info","ts":1660829269.0699186,"logger":"controllers.DevWorkspace","msg":"Reconciling Workspace","Request.Namespace":"d-sindhuja-che-a6q94i","Request.Name":"cbfsel-repo","devworkspace_id":"workspaceeb49234a892b48eb"}
{"level":"info","ts":1660829269.1085067,"logger":"controllers.DevWorkspace","msg":"Waiting on deployment to be ready","Request.Namespace":"d-sindhuja-che-a6q94i","Request.Name":"cbfsel-repo","devworkspace_id":"workspaceeb49234a892b48eb"}
{"level":"info","ts":1660829274.1175268,"logger":"controllers.DevWorkspace","msg":"Reconciling Workspace","Request.Namespace":"d-sindhuja-che-a6q94i","Request.Name":"cbfsel-repo","devworkspace_id":"workspaceeb49234a892b48eb"}
{"level":"info","ts":1660829274.1594837,"logger":"controllers.DevWorkspace","msg":"Waiting on deployment to be ready","Request.Namespace":"d-sindhuja-che-a6q94i","Request.Name":"cbfsel-repo","devworkspace_id":"workspaceeb49234a892b48eb"}
{"level":"info","ts":1660829279.1689675,"logger":"controllers.DevWorkspace","msg":"Reconciling Workspace","Request.Namespace":"d-sindhuja-che-a6q94i","Request.Name":"cbfsel-repo","devworkspace_id":"workspaceeb49234a892b48eb"}
{"level":"info","ts":1660829279.2289724,"logger":"controllers.DevWorkspace","msg":"Waiting on deployment to be ready","Request.Namespace":"d-sindhuja-che-a6q94i","Request.Name":"cbfsel-repo","devworkspace_id":"workspaceeb49234a892b48eb"}
{"level":"info","ts":1660829284.2445962,"logger":"controllers.DevWorkspace","msg":"Reconciling Workspace","Request.Namespace":"d-sindhuja-che-a6q94i","Request.Name":"cbfsel-repo","devworkspace_id":"workspaceeb49234a892b48eb"}
{"level":"info","ts":1660829284.2866073,"logger":"controllers.DevWorkspace","msg":"Waiting on deployment to be ready","Request.Namespace":"d-sindhuja-che-a6q94i","Request.Name":"cbfsel-repo","devworkspace_id":"workspaceeb49234a892b48eb"}
{"level":"info","ts":1660829289.2975821,"logger":"controllers.DevWorkspace","msg":"Reconciling Workspace","Request.Namespace":"d-sindhuja-che-a6q94i","Request.Name":"cbfsel-repo","devworkspace_id":"workspaceeb49234a892b48eb"}
{"level":"info","ts":1660829289.340491,"logger":"controllers.DevWorkspace","msg":"Waiting on deployment to be ready","Request.Namespace":"d-sindhuja-che-a6q94i","Request.Name":"cbfsel-repo","devworkspace_id":"workspaceeb49234a892b48eb"}
{"level":"info","ts":1660829294.350185,"logger":"controllers.DevWorkspace","msg":"Reconciling Workspace","Request.Namespace":"d-sindhuja-che-a6q94i","Request.Name":"cbfsel-repo","devworkspace_id":"workspaceeb49234a892b48eb"}
{"level":"info","ts":1660829294.3959272,"logger":"controllers.DevWorkspace","msg":"Waiting on deployment to be ready","Request.Namespace":"d-sindhuja-che-a6q94i","Request.Name":"cbfsel-repo","devworkspace_id":"workspaceeb49234a892b48eb"}
{"level":"info","ts":1660829299.406057,"logger":"controllers.DevWorkspace","msg":"Reconciling Workspace","Request.Namespace":"d-sindhuja-che-a6q94i","Request.Name":"cbfsel-repo","devworkspace_id":"workspaceeb49234a892b48eb"}
{"level":"info","ts":1660829299.4460104,"logger":"controllers.DevWorkspace","msg":"Waiting on deployment to be ready","Request.Namespace":"d-sindhuja-che-a6q94i","Request.Name":"cbfsel-repo","devworkspace_id":"workspaceeb49234a892b48eb"}
{"level":"info","ts":1660829299.4460633,"logger":"controllers.DevWorkspace","msg":"DevWorkspace failed to start: devworkspace failed to progress past phase 'Starting' for longer than timeout (5m)","Request.Namespace":"d-sindhuja-che-a6q94i","Request.Name":"cbfsel-repo","devworkspace_id":"workspaceeb49234a892b48eb"}
{"level":"info","ts":1660829299.4588716,"logger":"controllers.DevWorkspace","msg":"Reconciling Workspace","Request.Namespace":"d-sindhuja-che-a6q94i","Request.Name":"cbfsel-repo","devworkspace_id":"workspaceeb49234a892b48eb"}
{"level":"info","ts":1660829299.479444,"logger":"controllers.DevWorkspace","msg":"Reconciling Workspace","Request.Namespace":"d-sindhuja-che-a6q94i","Request.Name":"cbfsel-repo","devworkspace_id":"workspaceeb49234a892b48eb"}
{"level":"info","ts":1660829299.5057592,"logger":"controllers.DevWorkspace","msg":"Stopping workspace","Request.Namespace":"d-sindhuja-che-a6q94i","Request.Name":"cbfsel-repo","devworkspace_id":"workspaceeb49234a892b48eb"}
{"level":"info","ts":1660829299.546776,"logger":"controllers.DevWorkspace","msg":"Reconciling Workspace","Request.Namespace":"d-sindhuja-che-a6q94i","Request.Name":"cbfsel-repo","devworkspace_id":"workspaceeb49234a892b48eb"}
{"level":"info","ts":1660829299.5780475,"logger":"controllers.DevWorkspace","msg":"Reconciling Workspace","Request.Namespace":"d-sindhuja-che-a6q94i","Request.Name":"cbfsel-repo","devworkspace_id":"workspaceeb49234a892b48eb"}
{"level":"info","ts":1660829299.6191282,"logger":"controllers.DevWorkspace","msg":"Reconciling Workspace","Request.Namespace":"d-sindhuja-che-a6q94i","Request.Name":"cbfsel-repo","devworkspace_id":"workspaceeb49234a892b48eb"}
{"level":"info","ts":1660829299.6456137,"logger":"controllers.DevWorkspace","msg":"Reconciling Workspace","Request.Namespace":"d-sindhuja-che-a6q94i","Request.Name":"cbfsel-repo","devworkspace_id":"workspaceeb49234a892b48eb"}
{"level":"info","ts":1660829299.6579175,"logger":"controllers.DevWorkspace","msg":"Got conflict when trying to apply timing annotations to workspace","Request.Namespace":"d-sindhuja-che-a6q94i","Request.Name":"cbfsel-repo","devworkspace_id":"workspaceeb49234a892b48eb"}
{"level":"info","ts":1660829299.664098,"logger":"controllers.DevWorkspace","msg":"Error updating workspace status: Operation cannot be fulfilled on devworkspaces.workspace.devfile.io \"cbfsel-repo\": the object has been modified; please apply your changes to the latest version and try again","Request.Namespace":"d-sindhuja-che-a6q94i","Request.Name":"cbfsel-repo","devworkspace_id":"workspaceeb49234a892b48eb"}
{"level":"error","ts":1660829299.6641502,"logger":"controller-runtime.manager.controller.devworkspace","msg":"Reconciler error","reconciler group":"workspace.devfile.io","reconciler kind":"DevWorkspace","name":"cbfsel-repo","namespace":"d-sindhuja-che-a6q94i","error":"Operation cannot be fulfilled on devworkspaces.workspace.devfile.io \"cbfsel-repo\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:214"}
{"level":"info","ts":1660829299.669641,"logger":"controllers.DevWorkspace","msg":"Reconciling Workspace","Request.Namespace":"d-sindhuja-che-a6q94i","Request.Name":"cbfsel-repo","devworkspace_id":"workspaceeb49234a892b48eb"}
{"level":"info","ts":1660829307.5654163,"logger":"controllers.DevWorkspace","msg":"Reconciling Workspace","Request.Namespace":"d-sindhuja-che-a6q94i","Request.Name":"cbfsel-repo","devworkspace_id":"workspaceeb49234a892b48eb"}
below is the kubectl get events -n namespace showing failure messages while starting workspace pod
LAST SEEN TYPE REASON OBJECT MESSAGE
38m Normal ExternalProvisioning persistentvolumeclaim/claim-devworkspace waiting for a volume to be created, either by external provisioner "csi.vsphere.vmware.com" or manually created by system administrator
38m Normal Provisioning persistentvolumeclaim/claim-devworkspace External provisioner is provisioning volume for claim "d-sindhuja-che-a6q94i/claim-devworkspace"
38m Normal ProvisioningSucceeded persistentvolumeclaim/claim-devworkspace Successfully provisioned volume pvc-9d5082e1-7f0b-4ea2-bad0-6845ff945ced
12m Normal Scheduled pod/workspaceeb49234a892b48eb-856444584f-2j9xl Successfully assigned d-sindhuja-che-a6q94i/workspaceeb49234a892b48eb-856444584f-2j9xl to tkc-workers-wnnlk-b55cc6bdd-d6767
12m Normal Pulling pod/workspaceeb49234a892b48eb-856444584f-2j9xl Pulling image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next"
12m Normal Pulled pod/workspaceeb49234a892b48eb-856444584f-2j9xl Successfully pulled image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next" in 339.572392ms
12m Normal Created pod/workspaceeb49234a892b48eb-856444584f-2j9xl Created container remote-runtime-injector
12m Normal Started pod/workspaceeb49234a892b48eb-856444584f-2j9xl Started container remote-runtime-injector
12m Normal Pulling pod/workspaceeb49234a892b48eb-856444584f-2j9xl Pulling image "quay.io/devfile/project-clone:v0.15.2"
12m Normal Pulled pod/workspaceeb49234a892b48eb-856444584f-2j9xl Successfully pulled image "quay.io/devfile/project-clone:v0.15.2" in 342.804276ms
12m Normal Created pod/workspaceeb49234a892b48eb-856444584f-2j9xl Created container project-clone
12m Normal Started pod/workspaceeb49234a892b48eb-856444584f-2j9xl Started container project-clone
12m Normal Scheduled pod/workspaceeb49234a892b48eb-856444584f-2z9hf Successfully assigned d-sindhuja-che-a6q94i/workspaceeb49234a892b48eb-856444584f-2z9hf to tkc-workers-wnnlk-b55cc6bdd-d6767
12m Normal Pulling pod/workspaceeb49234a892b48eb-856444584f-2z9hf Pulling image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next"
12m Normal Pulled pod/workspaceeb49234a892b48eb-856444584f-2z9hf Successfully pulled image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next" in 521.508845ms
12m Normal Created pod/workspaceeb49234a892b48eb-856444584f-2z9hf Created container remote-runtime-injector
12m Normal Started pod/workspaceeb49234a892b48eb-856444584f-2z9hf Started container remote-runtime-injector
12m Normal Pulling pod/workspaceeb49234a892b48eb-856444584f-2z9hf Pulling image "quay.io/devfile/project-clone:v0.15.2"
12m Normal Pulled pod/workspaceeb49234a892b48eb-856444584f-2z9hf Successfully pulled image "quay.io/devfile/project-clone:v0.15.2" in 369.297726ms
12m Normal Created pod/workspaceeb49234a892b48eb-856444584f-2z9hf Created container project-clone
12m Normal Started pod/workspaceeb49234a892b48eb-856444584f-2z9hf Started container project-clone
14m Normal Scheduled pod/workspaceeb49234a892b48eb-856444584f-4269h Successfully assigned d-sindhuja-che-a6q94i/workspaceeb49234a892b48eb-856444584f-4269h to tkc-workers-wnnlk-b55cc6bdd-d6767
12m Warning FailedMount pod/workspaceeb49234a892b48eb-856444584f-4269h Unable to attach or mount volumes: unmounted volumes=[remote-endpoint automount-configmap-devworkspace-gitconfig kube-api-access-sgfcn claim-devworkspace automount-configmap-che-trusted-ca-certs workspace-metadata automount-secret-user-profile che-gateway automount-secret-devworkspace-merged-git-credentials automount-secret-che-server-cert], unattached volumes=[remote-endpoint automount-configmap-devworkspace-gitconfig kube-api-access-sgfcn claim-devworkspace automount-configmap-che-trusted-ca-certs workspace-metadata automount-secret-user-profile che-gateway automount-secret-devworkspace-merged-git-credentials automount-secret-che-server-cert]: timed out waiting for the condition
12m Normal Scheduled pod/workspaceeb49234a892b48eb-856444584f-4dt2t Successfully assigned d-sindhuja-che-a6q94i/workspaceeb49234a892b48eb-856444584f-4dt2t to tkc-workers-wnnlk-b55cc6bdd-d6767
12m Normal Pulling pod/workspaceeb49234a892b48eb-856444584f-4dt2t Pulling image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next"
12m Normal Pulled pod/workspaceeb49234a892b48eb-856444584f-4dt2t Successfully pulled image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next" in 355.33361ms
12m Normal Created pod/workspaceeb49234a892b48eb-856444584f-4dt2t Created container remote-runtime-injector
12m Normal Started pod/workspaceeb49234a892b48eb-856444584f-4dt2t Started container remote-runtime-injector
34m Normal Scheduled pod/workspaceeb49234a892b48eb-856444584f-55mh6 Successfully assigned d-sindhuja-che-a6q94i/workspaceeb49234a892b48eb-856444584f-55mh6 to tkc-workers-wnnlk-b55cc6bdd-d6767
34m Normal SuccessfulAttachVolume pod/workspaceeb49234a892b48eb-856444584f-55mh6 AttachVolume.Attach succeeded for volume "pvc-9d5082e1-7f0b-4ea2-bad0-6845ff945ced"
34m Normal Pulling pod/workspaceeb49234a892b48eb-856444584f-55mh6 Pulling image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next"
34m Normal Pulled pod/workspaceeb49234a892b48eb-856444584f-55mh6 Successfully pulled image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next" in 397.35665ms
34m Normal Created pod/workspaceeb49234a892b48eb-856444584f-55mh6 Created container remote-runtime-injector
34m Normal Started pod/workspaceeb49234a892b48eb-856444584f-55mh6 Started container remote-runtime-injector
34m Normal Pulling pod/workspaceeb49234a892b48eb-856444584f-55mh6 Pulling image "quay.io/devfile/project-clone:v0.15.2"
34m Normal Pulled pod/workspaceeb49234a892b48eb-856444584f-55mh6 Successfully pulled image "quay.io/devfile/project-clone:v0.15.2" in 349.773455ms
34m Normal Created pod/workspaceeb49234a892b48eb-856444584f-55mh6 Created container project-clone
34m Normal Started pod/workspaceeb49234a892b48eb-856444584f-55mh6 Started container project-clone
12m Normal Scheduled pod/workspaceeb49234a892b48eb-856444584f-7wh4h Successfully assigned d-sindhuja-che-a6q94i/workspaceeb49234a892b48eb-856444584f-7wh4h to tkc-workers-wnnlk-b55cc6bdd-d6767
12m Normal Pulling pod/workspaceeb49234a892b48eb-856444584f-7wh4h Pulling image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next"
12m Normal Pulled pod/workspaceeb49234a892b48eb-856444584f-7wh4h Successfully pulled image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next" in 347.474448ms
12m Normal Created pod/workspaceeb49234a892b48eb-856444584f-7wh4h Created container remote-runtime-injector
12m Normal Started pod/workspaceeb49234a892b48eb-856444584f-7wh4h Started container remote-runtime-injector
12m Normal Pulling pod/workspaceeb49234a892b48eb-856444584f-7wh4h Pulling image "quay.io/devfile/project-clone:v0.15.2"
12m Normal Pulled pod/workspaceeb49234a892b48eb-856444584f-7wh4h Successfully pulled image "quay.io/devfile/project-clone:v0.15.2" in 338.35664ms
12m Normal Created pod/workspaceeb49234a892b48eb-856444584f-7wh4h Created container project-clone
12m Normal Started pod/workspaceeb49234a892b48eb-856444584f-7wh4h Started container project-clone
36m Normal Scheduled pod/workspaceeb49234a892b48eb-856444584f-7wllq Successfully assigned d-sindhuja-che-a6q94i/workspaceeb49234a892b48eb-856444584f-7wllq to tkc-workers-wnnlk-b55cc6bdd-d6767
36m Normal Pulling pod/workspaceeb49234a892b48eb-856444584f-7wllq Pulling image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next"
36m Normal Pulled pod/workspaceeb49234a892b48eb-856444584f-7wllq Successfully pulled image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next" in 346.901236ms
36m Normal Created pod/workspaceeb49234a892b48eb-856444584f-7wllq Created container remote-runtime-injector
36m Normal Started pod/workspaceeb49234a892b48eb-856444584f-7wllq Started container remote-runtime-injector
36m Normal Pulling pod/workspaceeb49234a892b48eb-856444584f-7wllq Pulling image "quay.io/devfile/project-clone:v0.15.2"
36m Normal Pulled pod/workspaceeb49234a892b48eb-856444584f-7wllq Successfully pulled image "quay.io/devfile/project-clone:v0.15.2" in 334.85806ms
36m Normal Created pod/workspaceeb49234a892b48eb-856444584f-7wllq Created container project-clone
36m Normal Started pod/workspaceeb49234a892b48eb-856444584f-7wllq Started container project-clone
37m Normal Scheduled pod/workspaceeb49234a892b48eb-856444584f-8v6cx Successfully assigned d-sindhuja-che-a6q94i/workspaceeb49234a892b48eb-856444584f-8v6cx to tkc-workers-wnnlk-b55cc6bdd-d6767
37m Normal SuccessfulAttachVolume pod/workspaceeb49234a892b48eb-856444584f-8v6cx AttachVolume.Attach succeeded for volume "pvc-9d5082e1-7f0b-4ea2-bad0-6845ff945ced"
37m Normal Pulling pod/workspaceeb49234a892b48eb-856444584f-8v6cx Pulling image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next"
37m Normal Pulled pod/workspaceeb49234a892b48eb-856444584f-8v6cx Successfully pulled image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next" in 399.747138ms
37m Normal Created pod/workspaceeb49234a892b48eb-856444584f-8v6cx Created container remote-runtime-injector
37m Normal Started pod/workspaceeb49234a892b48eb-856444584f-8v6cx Started container remote-runtime-injector
37m Normal Pulling pod/workspaceeb49234a892b48eb-856444584f-8v6cx Pulling image "quay.io/devfile/project-clone:v0.15.2"
37m Normal Pulled pod/workspaceeb49234a892b48eb-856444584f-8v6cx Successfully pulled image "quay.io/devfile/project-clone:v0.15.2" in 352.797335ms
37m Normal Created pod/workspaceeb49234a892b48eb-856444584f-8v6cx Created container project-clone
37m Normal Started pod/workspaceeb49234a892b48eb-856444584f-8v6cx Started container project-clone
37m Normal Scheduled pod/workspaceeb49234a892b48eb-856444584f-9tzgn Successfully assigned d-sindhuja-che-a6q94i/workspaceeb49234a892b48eb-856444584f-9tzgn to tkc-workers-wnnlk-b55cc6bdd-d6767
37m Normal Pulling pod/workspaceeb49234a892b48eb-856444584f-9tzgn Pulling image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next"
37m Normal Pulled pod/workspaceeb49234a892b48eb-856444584f-9tzgn Successfully pulled image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next" in 351.501101ms
37m Normal Created pod/workspaceeb49234a892b48eb-856444584f-9tzgn Created container remote-runtime-injector
37m Normal Started pod/workspaceeb49234a892b48eb-856444584f-9tzgn Started container remote-runtime-injector
37m Normal Pulling pod/workspaceeb49234a892b48eb-856444584f-9tzgn Pulling image "quay.io/devfile/project-clone:v0.15.2"
37m Normal Pulled pod/workspaceeb49234a892b48eb-856444584f-9tzgn Successfully pulled image "quay.io/devfile/project-clone:v0.15.2" in 346.127765ms
37m Normal Created pod/workspaceeb49234a892b48eb-856444584f-9tzgn Created container project-clone
37m Normal Started pod/workspaceeb49234a892b48eb-856444584f-9tzgn Started container project-clone
15m Normal Scheduled pod/workspaceeb49234a892b48eb-856444584f-b79nn Successfully assigned d-sindhuja-che-a6q94i/workspaceeb49234a892b48eb-856444584f-b79nn to tkc-workers-wnnlk-b55cc6bdd-d6767
15m Normal Pulling pod/workspaceeb49234a892b48eb-856444584f-b79nn Pulling image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next"
15m Normal Pulled pod/workspaceeb49234a892b48eb-856444584f-b79nn Successfully pulled image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next" in 341.926397ms
15m Normal Created pod/workspaceeb49234a892b48eb-856444584f-b79nn Created container remote-runtime-injector
15m Normal Started pod/workspaceeb49234a892b48eb-856444584f-b79nn Started container remote-runtime-injector
15m Normal Pulling pod/workspaceeb49234a892b48eb-856444584f-b79nn Pulling image "quay.io/devfile/project-clone:v0.15.2"
15m Normal Pulled pod/workspaceeb49234a892b48eb-856444584f-b79nn Successfully pulled image "quay.io/devfile/project-clone:v0.15.2" in 325.354516ms
15m Normal Created pod/workspaceeb49234a892b48eb-856444584f-b79nn Created container project-clone
15m Normal Started pod/workspaceeb49234a892b48eb-856444584f-b79nn Started container project-clone
12m Warning FailedMount pod/workspaceeb49234a892b48eb-856444584f-b79nn Unable to attach or mount volumes: unmounted volumes=[remote-endpoint automount-secret-devworkspace-merged-git-credentials kube-api-access-5rzdr claim-devworkspace automount-secret-che-server-cert che-gateway automount-secret-user-profile workspace-metadata automount-configmap-devworkspace-gitconfig automount-configmap-che-trusted-ca-certs], unattached volumes=[remote-endpoint automount-secret-devworkspace-merged-git-credentials kube-api-access-5rzdr claim-devworkspace automount-secret-che-server-cert che-gateway automount-secret-user-profile workspace-metadata automount-configmap-devworkspace-gitconfig automount-configmap-che-trusted-ca-certs]: timed out waiting for the condition
15m Normal Scheduled pod/workspaceeb49234a892b48eb-856444584f-bscb6 Successfully assigned d-sindhuja-che-a6q94i/workspaceeb49234a892b48eb-856444584f-bscb6 to tkc-workers-wnnlk-b55cc6bdd-d6767
15m Normal Pulling pod/workspaceeb49234a892b48eb-856444584f-bscb6 Pulling image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next"
15m Normal Pulled pod/workspaceeb49234a892b48eb-856444584f-bscb6 Successfully pulled image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next" in 450.401917ms
15m Normal Created pod/workspaceeb49234a892b48eb-856444584f-bscb6 Created container remote-runtime-injector
15m Normal Started pod/workspaceeb49234a892b48eb-856444584f-bscb6 Started container remote-runtime-injector
16m Normal Scheduled pod/workspaceeb49234a892b48eb-856444584f-f8lsv Successfully assigned d-sindhuja-che-a6q94i/workspaceeb49234a892b48eb-856444584f-f8lsv to tkc-workers-wnnlk-b55cc6bdd-d6767
16m Normal SuccessfulAttachVolume pod/workspaceeb49234a892b48eb-856444584f-f8lsv AttachVolume.Attach succeeded for volume "pvc-9d5082e1-7f0b-4ea2-bad0-6845ff945ced"
14m Warning FailedMount pod/workspaceeb49234a892b48eb-856444584f-f8lsv Unable to attach or mount volumes: unmounted volumes=[automount-secret-user-profile che-gateway claim-devworkspace automount-configmap-devworkspace-gitconfig automount-secret-devworkspace-merged-git-credentials automount-secret-che-server-cert workspace-metadata kube-api-access-5mfm4 remote-endpoint automount-configmap-che-trusted-ca-certs], unattached volumes=[automount-secret-user-profile che-gateway claim-devworkspace automount-configmap-devworkspace-gitconfig automount-secret-devworkspace-merged-git-credentials automount-secret-che-server-cert workspace-metadata kube-api-access-5mfm4 remote-endpoint automount-configmap-che-trusted-ca-certs]: timed out waiting for the condition
12m Normal Scheduled pod/workspaceeb49234a892b48eb-856444584f-gl4hl Successfully assigned d-sindhuja-che-a6q94i/workspaceeb49234a892b48eb-856444584f-gl4hl to tkc-workers-wnnlk-b55cc6bdd-d6767
12m Normal Pulling pod/workspaceeb49234a892b48eb-856444584f-gl4hl Pulling image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next"
12m Normal Pulled pod/workspaceeb49234a892b48eb-856444584f-gl4hl Successfully pulled image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next" in 367.077215ms
12m Normal Created pod/workspaceeb49234a892b48eb-856444584f-gl4hl Created container remote-runtime-injector
12m Normal Started pod/workspaceeb49234a892b48eb-856444584f-gl4hl Started container remote-runtime-injector
12m Normal Pulling pod/workspaceeb49234a892b48eb-856444584f-gl4hl Pulling image "quay.io/devfile/project-clone:v0.15.2"
12m Normal Pulled pod/workspaceeb49234a892b48eb-856444584f-gl4hl Successfully pulled image "quay.io/devfile/project-clone:v0.15.2" in 341.145775ms
12m Normal Created pod/workspaceeb49234a892b48eb-856444584f-gl4hl Created container project-clone
12m Normal Started pod/workspaceeb49234a892b48eb-856444584f-gl4hl Started container project-clone
37m Normal Scheduled pod/workspaceeb49234a892b48eb-856444584f-k7m6p Successfully assigned d-sindhuja-che-a6q94i/workspaceeb49234a892b48eb-856444584f-k7m6p to tkc-workers-wnnlk-b55cc6bdd-d6767
37m Normal Pulling pod/workspaceeb49234a892b48eb-856444584f-k7m6p Pulling image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next"
37m Normal Pulled pod/workspaceeb49234a892b48eb-856444584f-k7m6p Successfully pulled image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next" in 356.596788ms
37m Normal Created pod/workspaceeb49234a892b48eb-856444584f-k7m6p Created container remote-runtime-injector
37m Normal Started pod/workspaceeb49234a892b48eb-856444584f-k7m6p Started container remote-runtime-injector
37m Normal Pulling pod/workspaceeb49234a892b48eb-856444584f-k7m6p Pulling image "quay.io/devfile/project-clone:v0.15.2"
37m Normal Pulled pod/workspaceeb49234a892b48eb-856444584f-k7m6p Successfully pulled image "quay.io/devfile/project-clone:v0.15.2" in 345.322594ms
37m Normal Created pod/workspaceeb49234a892b48eb-856444584f-k7m6p Created container project-clone
37m Normal Started pod/workspaceeb49234a892b48eb-856444584f-k7m6p Started container project-clone
36m Normal Scheduled pod/workspaceeb49234a892b48eb-856444584f-lt77t Successfully assigned d-sindhuja-che-a6q94i/workspaceeb49234a892b48eb-856444584f-lt77t to tkc-workers-wnnlk-b55cc6bdd-d6767
36m Normal Pulling pod/workspaceeb49234a892b48eb-856444584f-lt77t Pulling image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next"
36m Normal Pulled pod/workspaceeb49234a892b48eb-856444584f-lt77t Successfully pulled image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next" in 355.272188ms
36m Warning Failed pod/workspaceeb49234a892b48eb-856444584f-lt77t Error: stat /var/lib/kubelet/pods/d3ebe35a-5712-4159-90b1-489031bf6b88/volumes/kubernetes.io~configmap/automount-configmap-devworkspace-gitconfig: no such file or directory
15m Normal Scheduled pod/workspaceeb49234a892b48eb-856444584f-m754f Successfully assigned d-sindhuja-che-a6q94i/workspaceeb49234a892b48eb-856444584f-m754f to tkc-workers-wnnlk-b55cc6bdd-d6767
15m Normal Pulling pod/workspaceeb49234a892b48eb-856444584f-m754f Pulling image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next"
15m Normal Pulled pod/workspaceeb49234a892b48eb-856444584f-m754f Successfully pulled image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next" in 350.745007ms
15m Normal Created pod/workspaceeb49234a892b48eb-856444584f-m754f Created container remote-runtime-injector
15m Normal Started pod/workspaceeb49234a892b48eb-856444584f-m754f Started container remote-runtime-injector
15m Normal Pulling pod/workspaceeb49234a892b48eb-856444584f-m754f Pulling image "quay.io/devfile/project-clone:v0.15.2"
15m Normal Pulled pod/workspaceeb49234a892b48eb-856444584f-m754f Successfully pulled image "quay.io/devfile/project-clone:v0.15.2" in 357.574814ms
15m Normal Created pod/workspaceeb49234a892b48eb-856444584f-m754f Created container project-clone
15m Normal Started pod/workspaceeb49234a892b48eb-856444584f-m754f Started container project-clone
14m Normal Scheduled pod/workspaceeb49234a892b48eb-856444584f-n28s8 Successfully assigned d-sindhuja-che-a6q94i/workspaceeb49234a892b48eb-856444584f-n28s8 to tkc-workers-wnnlk-b55cc6bdd-d6767
14m Normal Pulling pod/workspaceeb49234a892b48eb-856444584f-n28s8 Pulling image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next"
14m Normal Pulled pod/workspaceeb49234a892b48eb-856444584f-n28s8 Successfully pulled image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next" in 350.132307ms
14m Normal Created pod/workspaceeb49234a892b48eb-856444584f-n28s8 Created container remote-runtime-injector
14m Normal Started pod/workspaceeb49234a892b48eb-856444584f-n28s8 Started container remote-runtime-injector
14m Normal Pulling pod/workspaceeb49234a892b48eb-856444584f-n28s8 Pulling image "quay.io/devfile/project-clone:v0.15.2"
14m Normal Pulled pod/workspaceeb49234a892b48eb-856444584f-n28s8 Successfully pulled image "quay.io/devfile/project-clone:v0.15.2" in 341.165608ms
14m Normal Created pod/workspaceeb49234a892b48eb-856444584f-n28s8 Created container project-clone
14m Normal Started pod/workspaceeb49234a892b48eb-856444584f-n28s8 Started container project-clone
14m Normal Scheduled pod/workspaceeb49234a892b48eb-856444584f-nrd6k Successfully assigned d-sindhuja-che-a6q94i/workspaceeb49234a892b48eb-856444584f-nrd6k to tkc-workers-wnnlk-b55cc6bdd-d6767
14m Normal Pulling pod/workspaceeb49234a892b48eb-856444584f-nrd6k Pulling image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next"
14m Normal Pulled pod/workspaceeb49234a892b48eb-856444584f-nrd6k Successfully pulled image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next" in 358.078334ms
14m Normal Created pod/workspaceeb49234a892b48eb-856444584f-nrd6k Created container remote-runtime-injector
14m Normal Started pod/workspaceeb49234a892b48eb-856444584f-nrd6k Started container remote-runtime-injector
14m Normal Pulling pod/workspaceeb49234a892b48eb-856444584f-nrd6k Pulling image "quay.io/devfile/project-clone:v0.15.2"
14m Normal Pulled pod/workspaceeb49234a892b48eb-856444584f-nrd6k Successfully pulled image "quay.io/devfile/project-clone:v0.15.2" in 391.75748ms
14m Normal Created pod/workspaceeb49234a892b48eb-856444584f-nrd6k Created container project-clone
14m Normal Started pod/workspaceeb49234a892b48eb-856444584f-nrd6k Started container project-clone
16m Normal Scheduled pod/workspaceeb49234a892b48eb-856444584f-q5gh9 Successfully assigned d-sindhuja-che-a6q94i/workspaceeb49234a892b48eb-856444584f-q5gh9 to tkc-workers-wnnlk-b55cc6bdd-d6767
15m Normal Pulling pod/workspaceeb49234a892b48eb-856444584f-q5gh9 Pulling image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next"
15m Normal Pulled pod/workspaceeb49234a892b48eb-856444584f-q5gh9 Successfully pulled image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next" in 423.440972ms
15m Normal Created pod/workspaceeb49234a892b48eb-856444584f-q5gh9 Created container remote-runtime-injector
15m Normal Started pod/workspaceeb49234a892b48eb-856444584f-q5gh9 Started container remote-runtime-injector
15m Normal Pulling pod/workspaceeb49234a892b48eb-856444584f-q5gh9 Pulling image "quay.io/devfile/project-clone:v0.15.2"
15m Normal Pulled pod/workspaceeb49234a892b48eb-856444584f-q5gh9 Successfully pulled image "quay.io/devfile/project-clone:v0.15.2" in 394.288373ms
15m Normal Created pod/workspaceeb49234a892b48eb-856444584f-q5gh9 Created container project-clone
15m Normal Started pod/workspaceeb49234a892b48eb-856444584f-q5gh9 Started container project-clone
36m Normal Scheduled pod/workspaceeb49234a892b48eb-856444584f-tbfnt Successfully assigned d-sindhuja-che-a6q94i/workspaceeb49234a892b48eb-856444584f-tbfnt to tkc-workers-wnnlk-b55cc6bdd-d6767
36m Normal Pulling pod/workspaceeb49234a892b48eb-856444584f-tbfnt Pulling image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next"
36m Normal Pulled pod/workspaceeb49234a892b48eb-856444584f-tbfnt Successfully pulled image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next" in 445.903266ms
36m Normal Created pod/workspaceeb49234a892b48eb-856444584f-tbfnt Created container remote-runtime-injector
36m Normal Started pod/workspaceeb49234a892b48eb-856444584f-tbfnt Started container remote-runtime-injector
36m Normal Pulling pod/workspaceeb49234a892b48eb-856444584f-tbfnt Pulling image "quay.io/devfile/project-clone:v0.15.2"
36m Normal Pulled pod/workspaceeb49234a892b48eb-856444584f-tbfnt Successfully pulled image "quay.io/devfile/project-clone:v0.15.2" in 373.632108ms
36m Normal Created pod/workspaceeb49234a892b48eb-856444584f-tbfnt Created container project-clone
36m Normal Started pod/workspaceeb49234a892b48eb-856444584f-tbfnt Started container project-clone
36m Normal SandboxChanged pod/workspaceeb49234a892b48eb-856444584f-tbfnt Pod sandbox changed, it will be killed and re-created.
36m Normal Pulled pod/workspaceeb49234a892b48eb-856444584f-tbfnt Successfully pulled image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next" in 362.563048ms
15m Normal Scheduled pod/workspaceeb49234a892b48eb-856444584f-tjqhp Successfully assigned d-sindhuja-che-a6q94i/workspaceeb49234a892b48eb-856444584f-tjqhp to tkc-workers-wnnlk-b55cc6bdd-d6767
14m Normal Pulling pod/workspaceeb49234a892b48eb-856444584f-tjqhp Pulling image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next"
14m Normal Pulled pod/workspaceeb49234a892b48eb-856444584f-tjqhp Successfully pulled image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next" in 367.062663ms
14m Normal Created pod/workspaceeb49234a892b48eb-856444584f-tjqhp Created container remote-runtime-injector
14m Normal Started pod/workspaceeb49234a892b48eb-856444584f-tjqhp Started container remote-runtime-injector
14m Normal Pulling pod/workspaceeb49234a892b48eb-856444584f-tjqhp Pulling image "quay.io/devfile/project-clone:v0.15.2"
14m Normal Pulled pod/workspaceeb49234a892b48eb-856444584f-tjqhp Successfully pulled image "quay.io/devfile/project-clone:v0.15.2" in 351.958852ms
14m Normal Created pod/workspaceeb49234a892b48eb-856444584f-tjqhp Created container project-clone
14m Warning Failed pod/workspaceeb49234a892b48eb-856444584f-tjqhp Error: failed to create containerd task: OCI runtime create failed: container_linux.go:364: creating new parent process caused: container_linux.go:2005: running lstat on namespace path "/proc/19665/ns/ipc" caused: lstat /proc/19665/ns/ipc: no such file or directory: unknown
14m Normal Scheduled pod/workspaceeb49234a892b48eb-856444584f-tt84x Successfully assigned d-sindhuja-che-a6q94i/workspaceeb49234a892b48eb-856444584f-tt84x to tkc-workers-wnnlk-b55cc6bdd-d6767
14m Normal SuccessfulAttachVolume pod/workspaceeb49234a892b48eb-856444584f-tt84x AttachVolume.Attach succeeded for volume "pvc-9d5082e1-7f0b-4ea2-bad0-6845ff945ced"
12m Warning FailedMount pod/workspaceeb49234a892b48eb-856444584f-tt84x Unable to attach or mount volumes: unmounted volumes=[claim-devworkspace remote-endpoint automount-secret-che-server-cert workspace-metadata automount-secret-user-profile che-gateway automount-secret-devworkspace-merged-git-credentials automount-configmap-devworkspace-gitconfig automount-configmap-che-trusted-ca-certs kube-api-access-zmvqs], unattached volumes=[claim-devworkspace remote-endpoint automount-secret-che-server-cert workspace-metadata automount-secret-user-profile che-gateway automount-secret-devworkspace-merged-git-credentials automount-configmap-devworkspace-gitconfig automount-configmap-che-trusted-ca-certs kube-api-access-zmvqs]: timed out waiting for the condition
14m Normal Scheduled pod/workspaceeb49234a892b48eb-856444584f-vq8dh Successfully assigned d-sindhuja-che-a6q94i/workspaceeb49234a892b48eb-856444584f-vq8dh to tkc-workers-wnnlk-b55cc6bdd-d6767
14m Normal Pulling pod/workspaceeb49234a892b48eb-856444584f-vq8dh Pulling image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next"
14m Normal Pulled pod/workspaceeb49234a892b48eb-856444584f-vq8dh Successfully pulled image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next" in 387.312813ms
14m Normal Created pod/workspaceeb49234a892b48eb-856444584f-vq8dh Created container remote-runtime-injector
14m Normal Started pod/workspaceeb49234a892b48eb-856444584f-vq8dh Started container remote-runtime-injector
14m Normal Pulling pod/workspaceeb49234a892b48eb-856444584f-vq8dh Pulling image "quay.io/devfile/project-clone:v0.15.2"
14m Normal Pulled pod/workspaceeb49234a892b48eb-856444584f-vq8dh Successfully pulled image "quay.io/devfile/project-clone:v0.15.2" in 387.398135ms
14m Warning Failed pod/workspaceeb49234a892b48eb-856444584f-vq8dh Error: failed to get sandbox container task: no running task found: task 3d154e239a6edcf33d5767e49c1fb00fabcef532af1e42105ba3efc75737420f not found: not found
14m Normal Scheduled pod/workspaceeb49234a892b48eb-856444584f-vs5zl Successfully assigned d-sindhuja-che-a6q94i/workspaceeb49234a892b48eb-856444584f-vs5zl to tkc-workers-wnnlk-b55cc6bdd-d6767
14m Normal Pulling pod/workspaceeb49234a892b48eb-856444584f-vs5zl Pulling image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next"
14m Normal Pulled pod/workspaceeb49234a892b48eb-856444584f-vs5zl Successfully pulled image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next" in 381.298757ms
14m Normal Created pod/workspaceeb49234a892b48eb-856444584f-vs5zl Created container remote-runtime-injector
14m Normal Started pod/workspaceeb49234a892b48eb-856444584f-vs5zl Started container remote-runtime-injector
14m Normal Pulling pod/workspaceeb49234a892b48eb-856444584f-vs5zl Pulling image "quay.io/devfile/project-clone:v0.15.2"
14m Normal Pulled pod/workspaceeb49234a892b48eb-856444584f-vs5zl Successfully pulled image "quay.io/devfile/project-clone:v0.15.2" in 365.850566ms
14m Normal Created pod/workspaceeb49234a892b48eb-856444584f-vs5zl Created container project-clone
14m Normal Started pod/workspaceeb49234a892b48eb-856444584f-vs5zl Started container project-clone
35m Normal Scheduled pod/workspaceeb49234a892b48eb-856444584f-wtknq Successfully assigned d-sindhuja-che-a6q94i/workspaceeb49234a892b48eb-856444584f-wtknq to tkc-workers-wnnlk-b55cc6bdd-d6767
35m Normal Pulling pod/workspaceeb49234a892b48eb-856444584f-wtknq Pulling image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next"
35m Normal Pulled pod/workspaceeb49234a892b48eb-856444584f-wtknq Successfully pulled image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next" in 421.452915ms
35m Normal Created pod/workspaceeb49234a892b48eb-856444584f-wtknq Created container remote-runtime-injector
35m Normal Started pod/workspaceeb49234a892b48eb-856444584f-wtknq Started container remote-runtime-injector
35m Normal Pulling pod/workspaceeb49234a892b48eb-856444584f-wtknq Pulling image "quay.io/devfile/project-clone:v0.15.2"
35m Normal Pulled pod/workspaceeb49234a892b48eb-856444584f-wtknq Successfully pulled image "quay.io/devfile/project-clone:v0.15.2" in 337.375129ms
35m Normal Created pod/workspaceeb49234a892b48eb-856444584f-wtknq Created container project-clone
35m Normal Started pod/workspaceeb49234a892b48eb-856444584f-wtknq Started container project-clone
33m Normal Scheduled pod/workspaceeb49234a892b48eb-856444584f-xbbsv Successfully assigned d-sindhuja-che-a6q94i/workspaceeb49234a892b48eb-856444584f-xbbsv to tkc-workers-wnnlk-b55cc6bdd-d6767
33m Normal Pulling pod/workspaceeb49234a892b48eb-856444584f-xbbsv Pulling image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next"
33m Normal Pulled pod/workspaceeb49234a892b48eb-856444584f-xbbsv Successfully pulled image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next" in 434.803688ms
33m Normal Created pod/workspaceeb49234a892b48eb-856444584f-xbbsv Created container remote-runtime-injector
33m Normal Started pod/workspaceeb49234a892b48eb-856444584f-xbbsv Started container remote-runtime-injector
12m Normal Scheduled pod/workspaceeb49234a892b48eb-856444584f-xhj6v Successfully assigned d-sindhuja-che-a6q94i/workspaceeb49234a892b48eb-856444584f-xhj6v to tkc-workers-wnnlk-b55cc6bdd-d6767
12m Normal Pulling pod/workspaceeb49234a892b48eb-856444584f-xhj6v Pulling image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next"
12m Normal Pulled pod/workspaceeb49234a892b48eb-856444584f-xhj6v Successfully pulled image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next" in 329.273227ms
12m Normal Created pod/workspaceeb49234a892b48eb-856444584f-xhj6v Created container remote-runtime-injector
12m Normal Started pod/workspaceeb49234a892b48eb-856444584f-xhj6v Started container remote-runtime-injector
12m Normal Pulling pod/workspaceeb49234a892b48eb-856444584f-xhj6v Pulling image "quay.io/devfile/project-clone:v0.15.2"
12m Normal Pulled pod/workspaceeb49234a892b48eb-856444584f-xhj6v Successfully pulled image "quay.io/devfile/project-clone:v0.15.2" in 387.489208ms
12m Normal Created pod/workspaceeb49234a892b48eb-856444584f-xhj6v Created container project-clone
12m Normal Started pod/workspaceeb49234a892b48eb-856444584f-xhj6v Started container project-clone
37m Normal SuccessfulCreate replicaset/workspaceeb49234a892b48eb-856444584f Created pod: workspaceeb49234a892b48eb-856444584f-8v6cx
37m Normal SuccessfulDelete replicaset/workspaceeb49234a892b48eb-856444584f Deleted pod: workspaceeb49234a892b48eb-856444584f-8v6cx
37m Normal SuccessfulCreate replicaset/workspaceeb49234a892b48eb-856444584f Created pod: workspaceeb49234a892b48eb-856444584f-9tzgn
37m Normal SuccessfulDelete replicaset/workspaceeb49234a892b48eb-856444584f Deleted pod: workspaceeb49234a892b48eb-856444584f-9tzgn
37m Normal SuccessfulCreate replicaset/workspaceeb49234a892b48eb-856444584f Created pod: workspaceeb49234a892b48eb-856444584f-k7m6p
36m Normal SuccessfulDelete replicaset/workspaceeb49234a892b48eb-856444584f Deleted pod: workspaceeb49234a892b48eb-856444584f-k7m6p
36m Normal SuccessfulCreate replicaset/workspaceeb49234a892b48eb-856444584f Created pod: workspaceeb49234a892b48eb-856444584f-tbfnt
36m Normal SuccessfulDelete replicaset/workspaceeb49234a892b48eb-856444584f Deleted pod: workspaceeb49234a892b48eb-856444584f-tbfnt
36m Normal SuccessfulCreate replicaset/workspaceeb49234a892b48eb-856444584f Created pod: workspaceeb49234a892b48eb-856444584f-lt77t
36m Normal SuccessfulDelete replicaset/workspaceeb49234a892b48eb-856444584f Deleted pod: workspaceeb49234a892b48eb-856444584f-lt77t
36m Normal SuccessfulCreate replicaset/workspaceeb49234a892b48eb-856444584f Created pod: workspaceeb49234a892b48eb-856444584f-7wllq
35m Normal SuccessfulDelete replicaset/workspaceeb49234a892b48eb-856444584f Deleted pod: workspaceeb49234a892b48eb-856444584f-7wllq
35m Normal SuccessfulCreate replicaset/workspaceeb49234a892b48eb-856444584f Created pod: workspaceeb49234a892b48eb-856444584f-wtknq
34m Normal SuccessfulDelete replicaset/workspaceeb49234a892b48eb-856444584f Deleted pod: workspaceeb49234a892b48eb-856444584f-wtknq
34m Normal SuccessfulCreate replicaset/workspaceeb49234a892b48eb-856444584f Created pod: workspaceeb49234a892b48eb-856444584f-55mh6
33m Normal SuccessfulDelete replicaset/workspaceeb49234a892b48eb-856444584f Deleted pod: workspaceeb49234a892b48eb-856444584f-55mh6
33m Normal SuccessfulCreate replicaset/workspaceeb49234a892b48eb-856444584f Created pod: workspaceeb49234a892b48eb-856444584f-xbbsv
33m Normal SuccessfulDelete replicaset/workspaceeb49234a892b48eb-856444584f Deleted pod: workspaceeb49234a892b48eb-856444584f-xbbsv
16m Normal SuccessfulCreate replicaset/workspaceeb49234a892b48eb-856444584f Created pod: workspaceeb49234a892b48eb-856444584f-f8lsv
16m Normal SuccessfulDelete replicaset/workspaceeb49234a892b48eb-856444584f Deleted pod: workspaceeb49234a892b48eb-856444584f-f8lsv
16m Normal SuccessfulCreate replicaset/workspaceeb49234a892b48eb-856444584f Created pod: workspaceeb49234a892b48eb-856444584f-q5gh9
15m Normal SuccessfulDelete replicaset/workspaceeb49234a892b48eb-856444584f Deleted pod: workspaceeb49234a892b48eb-856444584f-q5gh9
15m Normal SuccessfulCreate replicaset/workspaceeb49234a892b48eb-856444584f Created pod: workspaceeb49234a892b48eb-856444584f-bscb6
15m Normal SuccessfulDelete replicaset/workspaceeb49234a892b48eb-856444584f Deleted pod: workspaceeb49234a892b48eb-856444584f-bscb6
15m Normal SuccessfulCreate replicaset/workspaceeb49234a892b48eb-856444584f Created pod: workspaceeb49234a892b48eb-856444584f-m754f
15m Normal SuccessfulDelete replicaset/workspaceeb49234a892b48eb-856444584f Deleted pod: workspaceeb49234a892b48eb-856444584f-m754f
15m Normal SuccessfulCreate replicaset/workspaceeb49234a892b48eb-856444584f Created pod: workspaceeb49234a892b48eb-856444584f-b79nn
15m Normal SuccessfulDelete replicaset/workspaceeb49234a892b48eb-856444584f Deleted pod: workspaceeb49234a892b48eb-856444584f-b79nn
15m Normal SuccessfulCreate replicaset/workspaceeb49234a892b48eb-856444584f Created pod: workspaceeb49234a892b48eb-856444584f-tjqhp
14m Normal SuccessfulDelete replicaset/workspaceeb49234a892b48eb-856444584f Deleted pod: workspaceeb49234a892b48eb-856444584f-tjqhp
14m Normal SuccessfulCreate replicaset/workspaceeb49234a892b48eb-856444584f Created pod: workspaceeb49234a892b48eb-856444584f-n28s8
14m Normal SuccessfulDelete replicaset/workspaceeb49234a892b48eb-856444584f Deleted pod: workspaceeb49234a892b48eb-856444584f-n28s8
14m Normal SuccessfulCreate replicaset/workspaceeb49234a892b48eb-856444584f Created pod: workspaceeb49234a892b48eb-856444584f-nrd6k
14m Normal SuccessfulDelete replicaset/workspaceeb49234a892b48eb-856444584f Deleted pod: workspaceeb49234a892b48eb-856444584f-nrd6k
14m Normal SuccessfulCreate replicaset/workspaceeb49234a892b48eb-856444584f Created pod: workspaceeb49234a892b48eb-856444584f-vq8dh
14m Normal SuccessfulDelete replicaset/workspaceeb49234a892b48eb-856444584f Deleted pod: workspaceeb49234a892b48eb-856444584f-vq8dh
12m Normal SuccessfulCreate replicaset/workspaceeb49234a892b48eb-856444584f (combined from similar events): Created pod: workspaceeb49234a892b48eb-856444584f-2z9hf
13m Normal SuccessfulDelete replicaset/workspaceeb49234a892b48eb-856444584f (combined from similar events): Deleted pod: workspaceeb49234a892b48eb-856444584f-vs5zl
38m Warning FailedScheduling pod/workspaceeb49234a892b48eb-87f8964f9-7zjt2 0/4 nodes are available: 4 pod has unbound immediate PersistentVolumeClaims.
34m Normal Scheduled pod/workspaceeb49234a892b48eb-87f8964f9-ch744 Successfully assigned d-sindhuja-che-a6q94i/workspaceeb49234a892b48eb-87f8964f9-ch744 to tkc-workers-wnnlk-b55cc6bdd-d6767
34m Normal SuccessfulAttachVolume pod/workspaceeb49234a892b48eb-87f8964f9-ch744 AttachVolume.Attach succeeded for volume "pvc-9d5082e1-7f0b-4ea2-bad0-6845ff945ced"
32m Warning FailedMount pod/workspaceeb49234a892b48eb-87f8964f9-ch744 Unable to attach or mount volumes: unmounted volumes=[remote-endpoint workspace-metadata che-gateway automount-secret-user-profile kube-api-access-226t6 automount-configmap-devworkspace-gitconfig automount-configmap-che-trusted-ca-certs automount-secret-che-server-cert claim-devworkspace automount-secret-devworkspace-merged-git-credentials], unattached volumes=[remote-endpoint workspace-metadata che-gateway automount-secret-user-profile kube-api-access-226t6 automount-configmap-devworkspace-gitconfig automount-configmap-che-trusted-ca-certs automount-secret-che-server-cert claim-devworkspace automount-secret-devworkspace-merged-git-credentials]: timed out waiting for the condition
33m Normal Scheduled pod/workspaceeb49234a892b48eb-87f8964f9-fdczc Successfully assigned d-sindhuja-che-a6q94i/workspaceeb49234a892b48eb-87f8964f9-fdczc to tkc-workers-wnnlk-b55cc6bdd-d6767
33m Warning FailedMount pod/workspaceeb49234a892b48eb-87f8964f9-fdczc MountVolume.SetUp failed for volume "automount-secret-che-server-cert" : failed to sync secret cache: timed out waiting for the condition
33m Warning FailedMount pod/workspaceeb49234a892b48eb-87f8964f9-fdczc MountVolume.SetUp failed for volume "automount-configmap-devworkspace-gitconfig" : failed to sync configmap cache: timed out waiting for the condition
31m Warning FailedMount pod/workspaceeb49234a892b48eb-87f8964f9-fdczc Unable to attach or mount volumes: unmounted volumes=[workspace-metadata kube-api-access-8vv68 che-gateway automount-secret-devworkspace-merged-git-credentials automount-configmap-devworkspace-gitconfig claim-devworkspace automount-secret-user-profile automount-secret-che-server-cert remote-endpoint automount-configmap-che-trusted-ca-certs], unattached volumes=[workspace-metadata kube-api-access-8vv68 che-gateway automount-secret-devworkspace-merged-git-credentials automount-configmap-devworkspace-gitconfig claim-devworkspace automount-secret-user-profile automount-secret-che-server-cert remote-endpoint automount-configmap-che-trusted-ca-certs]: timed out waiting for the condition
33m Normal Scheduled pod/workspaceeb49234a892b48eb-87f8964f9-hfptc Successfully assigned d-sindhuja-che-a6q94i/workspaceeb49234a892b48eb-87f8964f9-hfptc to tkc-workers-wnnlk-b55cc6bdd-d6767
33m Normal SuccessfulAttachVolume pod/workspaceeb49234a892b48eb-87f8964f9-hfptc AttachVolume.Attach succeeded for volume "pvc-9d5082e1-7f0b-4ea2-bad0-6845ff945ced"
31m Warning FailedMount pod/workspaceeb49234a892b48eb-87f8964f9-hfptc Unable to attach or mount volumes: unmounted volumes=[automount-configmap-che-trusted-ca-certs che-gateway remote-endpoint automount-secret-devworkspace-merged-git-credentials automount-secret-che-server-cert workspace-metadata claim-devworkspace automount-configmap-devworkspace-gitconfig kube-api-access-kgt4g automount-secret-user-profile], unattached volumes=[automount-configmap-che-trusted-ca-certs che-gateway remote-endpoint automount-secret-devworkspace-merged-git-credentials automount-secret-che-server-cert workspace-metadata claim-devworkspace automount-configmap-devworkspace-gitconfig kube-api-access-kgt4g automount-secret-user-profile]: timed out waiting for the condition
14m Normal Scheduled pod/workspaceeb49234a892b48eb-87f8964f9-thk9z Successfully assigned d-sindhuja-che-a6q94i/workspaceeb49234a892b48eb-87f8964f9-thk9z to tkc-workers-wnnlk-b55cc6bdd-d6767
14m Normal Pulling pod/workspaceeb49234a892b48eb-87f8964f9-thk9z Pulling image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next"
14m Normal Pulled pod/workspaceeb49234a892b48eb-87f8964f9-thk9z Successfully pulled image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next" in 383.576145ms
14m Normal Created pod/workspaceeb49234a892b48eb-87f8964f9-thk9z Created container remote-runtime-injector
14m Normal Started pod/workspaceeb49234a892b48eb-87f8964f9-thk9z Started container remote-runtime-injector
14m Normal Pulling pod/workspaceeb49234a892b48eb-87f8964f9-thk9z Pulling image "quay.io/devfile/project-clone:v0.15.2"
14m Normal Pulled pod/workspaceeb49234a892b48eb-87f8964f9-thk9z Successfully pulled image "quay.io/devfile/project-clone:v0.15.2" in 355.726024ms
14m Normal Created pod/workspaceeb49234a892b48eb-87f8964f9-thk9z Created container project-clone
14m Normal Started pod/workspaceeb49234a892b48eb-87f8964f9-thk9z Started container project-clone
13m Normal Scheduled pod/workspaceeb49234a892b48eb-87f8964f9-vz6gc Successfully assigned d-sindhuja-che-a6q94i/workspaceeb49234a892b48eb-87f8964f9-vz6gc to tkc-workers-wnnlk-b55cc6bdd-d6767
13m Normal Pulling pod/workspaceeb49234a892b48eb-87f8964f9-vz6gc Pulling image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next"
13m Normal Pulled pod/workspaceeb49234a892b48eb-87f8964f9-vz6gc Successfully pulled image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next" in 416.800862ms
13m Normal Created pod/workspaceeb49234a892b48eb-87f8964f9-vz6gc Created container remote-runtime-injector
13m Normal Started pod/workspaceeb49234a892b48eb-87f8964f9-vz6gc Started container remote-runtime-injector
13m Normal Pulling pod/workspaceeb49234a892b48eb-87f8964f9-vz6gc Pulling image "quay.io/devfile/project-clone:v0.15.2"
13m Normal Pulled pod/workspaceeb49234a892b48eb-87f8964f9-vz6gc Successfully pulled image "quay.io/devfile/project-clone:v0.15.2" in 348.820088ms
13m Normal Created pod/workspaceeb49234a892b48eb-87f8964f9-vz6gc Created container project-clone
13m Normal Started pod/workspaceeb49234a892b48eb-87f8964f9-vz6gc Started container project-clone
32m Normal Scheduled pod/workspaceeb49234a892b48eb-87f8964f9-w5xc6 Successfully assigned d-sindhuja-che-a6q94i/workspaceeb49234a892b48eb-87f8964f9-w5xc6 to tkc-workers-wnnlk-b55cc6bdd-d6767
32m Normal SuccessfulAttachVolume pod/workspaceeb49234a892b48eb-87f8964f9-w5xc6 AttachVolume.Attach succeeded for volume "pvc-9d5082e1-7f0b-4ea2-bad0-6845ff945ced"
30m Warning FailedMount pod/workspaceeb49234a892b48eb-87f8964f9-w5xc6 Unable to attach or mount volumes: unmounted volumes=[remote-endpoint che-gateway automount-secret-devworkspace-merged-git-credentials automount-configmap-che-trusted-ca-certs automount-secret-user-profile workspace-metadata claim-devworkspace automount-secret-che-server-cert kube-api-access-d54l7 automount-configmap-devworkspace-gitconfig], unattached volumes=[remote-endpoint che-gateway automount-secret-devworkspace-merged-git-credentials automount-configmap-che-trusted-ca-certs automount-secret-user-profile workspace-metadata claim-devworkspace automount-secret-che-server-cert kube-api-access-d54l7 automount-configmap-devworkspace-gitconfig]: timed out waiting for the condition
38m Normal SuccessfulCreate replicaset/workspaceeb49234a892b48eb-87f8964f9 Created pod: workspaceeb49234a892b48eb-87f8964f9-7zjt2
38m Normal SuccessfulDelete replicaset/workspaceeb49234a892b48eb-87f8964f9 Deleted pod: workspaceeb49234a892b48eb-87f8964f9-7zjt2
34m Normal SuccessfulCreate replicaset/workspaceeb49234a892b48eb-87f8964f9 Created pod: workspaceeb49234a892b48eb-87f8964f9-ch744
34m Normal SuccessfulDelete replicaset/workspaceeb49234a892b48eb-87f8964f9 Deleted pod: workspaceeb49234a892b48eb-87f8964f9-ch744
33m Normal SuccessfulCreate replicaset/workspaceeb49234a892b48eb-87f8964f9 Created pod: workspaceeb49234a892b48eb-87f8964f9-fdczc
33m Normal SuccessfulDelete replicaset/workspaceeb49234a892b48eb-87f8964f9 Deleted pod: workspaceeb49234a892b48eb-87f8964f9-fdczc
33m Normal SuccessfulCreate replicaset/workspaceeb49234a892b48eb-87f8964f9 Created pod: workspaceeb49234a892b48eb-87f8964f9-hfptc
32m Normal SuccessfulDelete replicaset/workspaceeb49234a892b48eb-87f8964f9 Deleted pod: workspaceeb49234a892b48eb-87f8964f9-hfptc
32m Normal SuccessfulCreate replicaset/workspaceeb49234a892b48eb-87f8964f9 Created pod: workspaceeb49234a892b48eb-87f8964f9-w5xc6
32m Normal SuccessfulDelete replicaset/workspaceeb49234a892b48eb-87f8964f9 Deleted pod: workspaceeb49234a892b48eb-87f8964f9-w5xc6
14m Normal SuccessfulCreate replicaset/workspaceeb49234a892b48eb-87f8964f9 Created pod: workspaceeb49234a892b48eb-87f8964f9-thk9z
14m Normal SuccessfulDelete replicaset/workspaceeb49234a892b48eb-87f8964f9 Deleted pod: workspaceeb49234a892b48eb-87f8964f9-thk9z
13m Normal SuccessfulCreate replicaset/workspaceeb49234a892b48eb-87f8964f9 Created pod: workspaceeb49234a892b48eb-87f8964f9-vz6gc
12m Normal SuccessfulDelete replicaset/workspaceeb49234a892b48eb-87f8964f9 Deleted pod: workspaceeb49234a892b48eb-87f8964f9-vz6gc
13m Warning FailedToUpdateEndpoint endpoints/workspaceeb49234a892b48eb-service Failed to update endpoint d-sindhuja-che-a6q94i/workspaceeb49234a892b48eb-service: Operation cannot be fulfilled on endpoints "workspaceeb49234a892b48eb-service": the object has been modified; please apply your changes to the latest version and try again
33m Normal ScalingReplicaSet deployment/workspaceeb49234a892b48eb Scaled up replica set workspaceeb49234a892b48eb-87f8964f9 to 1
12m Normal ScalingReplicaSet deployment/workspaceeb49234a892b48eb Scaled down replica set workspaceeb49234a892b48eb-87f8964f9 to 0
16m Normal ScalingReplicaSet deployment/workspaceeb49234a892b48eb Scaled up replica set workspaceeb49234a892b48eb-856444584f to 1
16m Normal ScalingReplicaSet deployment/workspaceeb49234a892b48eb Scaled down replica set workspaceeb49234a892b48eb-856444584f to 0
Maybe switching to per-workspace pvc strategy can help here:
kubectl patch checluster/eclipse-che --patch '{"spec": {"devEnvironments": {"storage": {"pvcStrategy": "per-workspace"}}}}' --type=merge -n eclipse-che
See the corresponding PR https://github.com/devfile/devworkspace-operator/pull/818
@AObuchow
Could you confirm that setting per-workspace in CheCluster CR works?
Or should we wait for some PR is merged?
@AObuchow Could you confirm that setting
per-workspacein CheCluster CR works? Or should we wait for some PR is merged?
Unfortunately, setting per-workspace in CheCluster CR doesn't currently have any effect on DevWorkspace-Operator, and a future PR is required for this.
However, for the time being, the perWorkspace PVC strategy should be able to be enabled by adding the controller.devfile.io/storage-type attribute to a devfile and setting it to per-workspace. E.g:
schemaVersion: 2.1.0
metadata:
name: cbfsel-repo
+ attributes:
+ controller.devfile.io/storage-type: per-workspace
projects:
- name: cbfsel-project
git:
checkoutFrom:
revision: master
remotes:
origin: https://gitlab.com/dchelladurai/cbf-sel.git
components:
- container:
image: 'artfact-prd/qedocker/eclipseche/customopenjdk8:v10'
memoryLimit: 4G
name: javacontainer
commands:
- exec:
commandLine: mvn clean package -DskipTests
component: javacontainer
group:
isDefault: true
kind: build
label: 'build project using maven'
workingDir: '${PROJECT_SOURCE}'
id: mvnpackage
Here's the documentation for storage configuration in DevWorkspace-Operator
Lastly for reference, here's another devfile using the controller.devfile.io/storage-type attribute, as an example (though it's set to ephemeral rather than per-workspace.
I hope this helps :)
~~Edit: I was having issues getting this to work in my own testing on our dogfooding instance of Eclipse Che, so it's possible this advice is not useful.~~ Edit 2: Turns out the issue was unrelated, and I was able to get this attribute working in this devfile
@AObuchow @tolusha
i receive below error message after adding per-workspace in my devfile.yaml.
chectl version
chectl/0.0.20220809-next.c9f4ba1 darwin-x64 node-v16.13.2
below is my devfile.yaml
schemaVersion: 2.1.0
metadata:
name: cbfsel-repo
attributes:
controller.devfile.io/storage-type: per-workspace
projects:
- name: cbfsel-project
git:
checkoutFrom:
revision: master
remotes:
origin: https://gitlab.eng.com/dchelladurai/cbf-sel.git
components:
- container:
image: 'artfact-prd.com:5001/qedocker/eclipseche/customopenjdk8:v10'
memoryLimit: 4G
name: javacontainer
commands:
- exec:
commandLine: mvn clean package -DskipTests
component: javacontainer
group:
isDefault: true
kind: build
label: 'build project using maven'
workingDir: '${PROJECT_SOURCE}'
id: mvnpackage
please let me know how to fix this
Which is devworkspace controller image used?
kubectl get deployment devworkspace-controller-manager -n devworkspace-controller -o="jsonpath={.spec.template.spec.containers[0].image}"
I see. That's a bit outdated. The latest version is v0.15.3
It is supposed that the latest che-operator deploys the latest version of devworkspace operator.
Could you simply restart che-operator pods and then waits until it completely reconciled.
Does it update devworkspace resources after restart ?
@tolusha
i scaled down and scaledup by che-operator deployment. The che-operator deployment was successful. but it didnot update the resources in devworkspace-controller namespace.
That's strange. What is your che-operator image:
kubectl get deployment che-operator -n eclipse-che -o="jsonpath={.spec.template.spec.containers[0].image}
@tolusha
my che-operator image is quay.io/eclipse/che-operator:next
i have installed my eclipse-che using chectl version chectl/0.0.20220809-next.c9f4ba1 darwin-x64 node-v16.13.2
chectl server:deploy --che-operator-cr-patch-yaml=/Users/divine/Documents/office_tasks/TAP-4540/terraformCheJune17/eclipseche_yaml/che-operator-cr-patch.yaml --platform=k8s --installer=operator --debug --k8spoderrorrechecktimeout=1500000 --domain=eclipseche-dchelladurai-chejune15.calatrava.vmware.com --k8spodreadytimeout=1500000
Could you also check as well
kubectl get namespace devworkspace-controller -o jsonpath="{.metadata.annotations}"
@tolusha
output for kubectl get namespace devworkspace-controller -o jsonpath="{.metadata.annotations}" is below
{"che.eclipse.org/namespace":"eclipse-che"}
I am out of ideas. Could you check this please?
kubectl get deployment che-operator -n eclipse-che -o json | jq -r '.spec.template.spec.containers[0].env[] | select(.name=="RELATED_IMAGE_devworkspace_controller")'
kubectl get checluster eclipse-che -n eclipse-che -o="jsonpath={.spec.components.devWorkspace}"
@tolusha
i have added the full output of che-operator and checluster . please verify
kubectl get deployment che-operator -n eclipse-che -o yaml
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "1"
creationTimestamp: "2022-08-16T08:23:57Z"
generation: 3
labels:
app: che-operator
app.kubernetes.io/component: che-operator
app.kubernetes.io/instance: che
app.kubernetes.io/name: che
app.kubernetes.io/part-of: che.eclipse.org
name: che-operator
namespace: eclipse-che
resourceVersion: "31483987"
uid: 1f4c2c99-4871-410a-bab9-88abc13da6aa
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: che-operator
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app: che-operator
app.kubernetes.io/component: che-operator
app.kubernetes.io/instance: che
app.kubernetes.io/name: che
app.kubernetes.io/part-of: che.eclipse.org
spec:
containers:
- args:
- --leader-elect
command:
- /manager
env:
- name: WATCH_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: OPERATOR_NAME
value: che-operator
- name: CHE_VERSION
value: next
- name: RELATED_IMAGE_che_server
value: quay.io/eclipse/che-server:next
- name: RELATED_IMAGE_dashboard
value: quay.io/eclipse/che-dashboard:next
- name: RELATED_IMAGE_plugin_registry
value: quay.io/eclipse/che-plugin-registry:next
- name: RELATED_IMAGE_devfile_registry
value: quay.io/eclipse/che-devfile-registry:next
- name: RELATED_IMAGE_che_tls_secrets_creation_job
value: quay.io/eclipse/che-tls-secret-creator:alpine-01a4c34
- name: RELATED_IMAGE_postgres
value: quay.io/eclipse/che--centos--postgresql-96-centos7:9.6-b681d78125361519180a6ac05242c296f8906c11eab7e207b5ca9a89b6344392
- name: RELATED_IMAGE_postgres_13_3
value: quay.io/eclipse/che--centos--postgresql-13-centos7:1-71b24684d64da46f960682cc4216222a7e4ed8b1a31dd5a865b3e71afdea20d2
- name: RELATED_IMAGE_single_host_gateway
value: quay.io/eclipse/che--traefik:v2.8.1-4e52a5e2495484f5e19a49edfd2f652b0bce7b3603fa0df545ed90168ffae1c3
- name: RELATED_IMAGE_single_host_gateway_config_sidecar
value: quay.io/che-incubator/configbump:0.1.4
- name: RELATED_IMAGE_devworkspace_controller
value: quay.io/devfile/devworkspace-controller:v0.15.2
- name: RELATED_IMAGE_gateway_authentication_sidecar
value: quay.io/openshift/origin-oauth-proxy:4.7
- name: RELATED_IMAGE_gateway_authorization_sidecar
value: quay.io/openshift/origin-kube-rbac-proxy:4.7
- name: RELATED_IMAGE_gateway_authentication_sidecar_k8s
value: quay.io/oauth2-proxy/oauth2-proxy:v7.2.0
- name: RELATED_IMAGE_gateway_authorization_sidecar_k8s
value: quay.io/brancz/kube-rbac-proxy:v0.11.0
- name: RELATED_IMAGE_gateway_header_sidecar
value: quay.io/che-incubator/header-rewrite-proxy:latest
- name: CHE_FLAVOR
value: che
- name: CONSOLE_LINK_NAME
value: che
- name: CONSOLE_LINK_DISPLAY_NAME
value: Eclipse Che
- name: CONSOLE_LINK_SECTION
value: Red Hat Applications
- name: CONSOLE_LINK_IMAGE
value: /dashboard/assets/branding/loader.svg
- name: MAX_CONCURRENT_RECONCILES
value: "1"
- name: ADD_COMPONENT_READINESS_INIT_CONTAINERS
value: "false"
image: quay.io/eclipse/che-operator:next
imagePullPolicy: Always
livenessProbe:
failureThreshold: 10
httpGet:
path: /healthz
port: 6789
scheme: HTTP
initialDelaySeconds: 15
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
name: che-operator
ports:
- containerPort: 9443
name: webhook-server
protocol: TCP
- containerPort: 60000
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 10
httpGet:
path: /readyz
port: 6789
scheme: HTTP
initialDelaySeconds: 5
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
resources:
limits:
cpu: 500m
memory: 256Mi
requests:
cpu: 100m
memory: 64Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
privileged: false
readOnlyRootFilesystem: false
runAsNonRoot: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /tmp/k8s-webhook-server/serving-certs
name: webhook-tls-certs
readOnly: true
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
fsGroup: 1724
runAsUser: 1724
serviceAccount: che-operator
serviceAccountName: che-operator
terminationGracePeriodSeconds: 20
volumes:
- name: webhook-tls-certs
secret:
defaultMode: 420
secretName: che-operator-webhook-server-cert
status:
availableReplicas: 1
conditions:
- lastTransitionTime: "2022-08-16T08:23:57Z"
lastUpdateTime: "2022-08-16T08:24:18Z"
message: ReplicaSet "che-operator-c489c48f4" has successfully progressed.
reason: NewReplicaSetAvailable
status: "True"
type: Progressing
- lastTransitionTime: "2022-08-24T13:32:40Z"
lastUpdateTime: "2022-08-24T13:32:40Z"
message: Deployment has minimum availability.
reason: MinimumReplicasAvailable
status: "True"
type: Available
observedGeneration: 3
readyReplicas: 1
replicas: 1
updatedReplicas: 1
kubectl get checluster eclipse-che -n eclipse-che -o yaml
apiVersion: org.eclipse.che/v2
kind: CheCluster
metadata:
creationTimestamp: "2022-08-16T08:25:32Z"
finalizers:
- checluster.che.eclipse.org
- cheGateway.clusterpermissions.finalizers.che.eclipse.org
- cheWorkspaces.clusterpermissions.finalizers.che.eclipse.org
- namespaces-editor.permissions.finalizers.che.eclipse.org
- devWorkspace.permissions.finalizers.che.eclipse.org
- dashboard.clusterpermissions.finalizers.che.eclipse.org
generation: 2
name: eclipse-che
namespace: eclipse-che
resourceVersion: "27746407"
uid: b29d5725-05b4-4d6f-b97b-476ff3079798
spec:
components:
cheServer:
debug: true
extraProperties:
CHE_INFRA_KUBERNETES_PVC_QUANTITY: 5Gi
CHE_INFRA_KUBERNETES_WORKSPACE__START__TIMEOUT__MIN: "15"
logLevel: INFO
dashboard: {}
database:
credentialsSecretName: postgres-credentials
externalDb: true
postgresDb: dbche
postgresHostName: sc2-10-186-67-195.eng.vmware.com
postgresPort: "5432"
pvc:
claimSize: 1Gi
devWorkspace: {}
devfileRegistry: {}
imagePuller:
enable: false
spec: {}
metrics:
enable: true
pluginRegistry:
deployment:
containers:
- image: artfact-prd.vmware.com:5001/qedocker/eclipseche/che-plugin-registry:v7
containerRegistry: {}
devEnvironments:
defaultNamespace:
template: <username>-che
nodeSelector:
kubernetes.io/hostname: tkc-workers-wnnlk-b55cc6bdd-d6767
secondsOfInactivityBeforeIdling: 1800
secondsOfRunBeforeIdling: -1
storage:
pvcStrategy: common
networking:
auth:
gateway:
configLabels:
app: che
component: che-gateway-config
identityProviderURL: https://dex-dchelladurai-chejune15.calatrava.vmware.com
oAuthClientName: eclipse-che
oAuthSecret: ZXhhbXBsZS1hcHAtc2VjcmV0
domain: eclipseche-dchelladurai-chejune15.calatrava.vmware.com
tlsSecretName: che-tls
status:
chePhase: Active
cheURL: https://eclipseche-dchelladurai-chejune15.calatrava.vmware.com
cheVersion: next
devfileRegistryURL: https://eclipseche-dchelladurai-chejune15.calatrava.vmware.com/devfile-registry
gatewayPhase: Established
pluginRegistryURL: https://eclipseche-dchelladurai-chejune15.calatrava.vmware.com/plugin-registry/v3
workspaceBaseDomain: eclipseche-dchelladurai-chejune15.calatrava.vmware.com
@tolusha should i upgrade eclipse-che to fix this issue?
currently i'm using chectl/0.0.20220809-next.c9f4ba1 darwin-x64 node-v16.13.2
i'm facing this issue for past few months, please help me to fix this issue
Hello, I am also facing this issue but little different. I am trying to create the sample workspace and it is not taking the storageclass name i passed in the checluster configuration. so getting below error. please help.
0s Normal FailedBinding persistentvolumeclaim/claim-devworkspace no persistent volumes available for this claim and no storage class is set
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
postgres-pv-volume 1Gi RWO Retain Bound eclipse-che/postgres-data postgres-storage 14m
workspace-pv-volume 20Gi RWO Retain Available workspace-storage 24m
I have eclipse-che running on 2 nodes K8s cluster with dex as OIDC.
Issues go stale after 180 days of inactivity. lifecycle/stale issues rot after an additional 7 days of inactivity and eventually close.
Mark the issue as fresh with /remove-lifecycle stale in a new comment.
If this issue is safe to close now please do so.
Moderators: Add lifecycle/frozen label to avoid stale mode.
