gcs
gcs copied to clipboard
Published
20 hours ago •
gluster
gluster
etcd pod in Error state for 10 mins after a node reboot
- Created 50 PVC parallel requests.
- While the pvc are getting processed , rebooted second node from the 3 node GCS cluster.
[vagrant@kube1 ~]$ kubectl get pods -n gcs
NAME READY STATUS RESTARTS AGE
csi-attacher-glusterfsplugin-0 2/2 Running 0 12m
csi-nodeplugin-glusterfsplugin-69jvq 2/2 Running 0 12m
csi-nodeplugin-glusterfsplugin-bf5kx 2/2 Running 0 12m
csi-nodeplugin-glusterfsplugin-n7wzw 0/2 Error 0 12m
csi-provisioner-glusterfsplugin-0 3/3 Running 0 12m
etcd-9dnxxnvhq2 1/1 Running 0 18m
etcd-9wz7txxx59 1/1 Running 0 16m
etcd-b92cpssvmz 0/1 Error 0 17m
etcd-operator-7cb5bd459b-mczrk 1/1 Running 0 18m
gluster-kube1-0 1/1 Running 1 16m
gluster-kube2-0 0/1 Error 1 16m
gluster-kube3-0 1/1 Running 1 16m
- Observing the pvc stuck at 38 for the time being.
[vagrant@kube1 ~]$ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-d92e016d-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc0 glusterfs-csi 116s
pvc-d959ddbe-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc1 glusterfs-csi 115s
pvc-d9860619-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc2 glusterfs-csi 2m19s
pvc-d9b5e137-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc3 glusterfs-csi 2m19s
pvc-da1ac34c-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc5 glusterfs-csi 2m18s
pvc-da510fc2-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc6 glusterfs-csi 2m18s
pvc-db002280-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc9 glusterfs-csi 2m17s
pvc-db342a77-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc10 glusterfs-csi 2m15s
pvc-db66d2fe-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc11 glusterfs-csi 119s
pvc-db9bcff5-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc12 glusterfs-csi 2m16s
pvc-dbce239d-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc13 glusterfs-csi 114s
pvc-dc04ef7a-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc14 glusterfs-csi 2m15s
pvc-dc39d4bc-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc15 glusterfs-csi 2m14s
pvc-dca4b450-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc17 glusterfs-csi 2m13s
pvc-dcddb56f-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc18 glusterfs-csi 2m12s
pvc-dd136269-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc19 glusterfs-csi 2m12s
pvc-dd459ffd-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc20 glusterfs-csi 118s
pvc-ddaed68a-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc22 glusterfs-csi 116s
pvc-dde6b8bb-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc23 glusterfs-csi 2m11s
pvc-de4d84fc-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc25 glusterfs-csi 2m9s
pvc-de8758b5-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc26 glusterfs-csi 119s
pvc-dec2600c-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc27 glusterfs-csi 2m9s
pvc-def3a474-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc28 glusterfs-csi 2m9s
pvc-df2e25d9-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc29 glusterfs-csi 2m8s
pvc-df64fa77-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc30 glusterfs-csi 2m6s
pvc-df99124a-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc31 glusterfs-csi 2m6s
pvc-dfce5481-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc32 glusterfs-csi 2m5s
pvc-e0066f0b-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc33 glusterfs-csi 2m4s
pvc-e0431efb-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc34 glusterfs-csi 2m3s
pvc-e0925da6-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc35 glusterfs-csi 2m3s
pvc-e0c314f3-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc36 glusterfs-csi 2m2s
pvc-e0f896db-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc37 glusterfs-csi 2m1s
pvc-e12be8b5-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc38 glusterfs-csi 119s
- Eventually the gd2 pods and csi nodeplugin came up and running, but the etcd remain in the same state.
[vagrant@kube1 ~]$ kubectl get pods -n gcs
NAME READY STATUS RESTARTS AGE
csi-attacher-glusterfsplugin-0 2/2 Running 0 15m
csi-nodeplugin-glusterfsplugin-69jvq 2/2 Running 0 15m
csi-nodeplugin-glusterfsplugin-bf5kx 2/2 Running 0 15m
csi-nodeplugin-glusterfsplugin-n7wzw 2/2 Running 2 15m
csi-provisioner-glusterfsplugin-0 3/3 Running 0 15m
etcd-9dnxxnvhq2 1/1 Running 0 21m
etcd-9wz7txxx59 1/1 Running 0 19m
etcd-b92cpssvmz 0/1 Error 0 20m
etcd-operator-7cb5bd459b-mczrk 1/1 Running 0 21m
gluster-kube1-0 1/1 Running 1 19m
gluster-kube2-0 1/1 Running 3 19m
gluster-kube3-0 1/1 Running 1 19m
- Wait for 10 mins, the etcd pod gets killed and a new pod is spun
[vagrant@kube1 ~]$ kubectl get pods -n gcs
NAME READY STATUS RESTARTS AGE
csi-attacher-glusterfsplugin-0 2/2 Running 0 22m
csi-nodeplugin-glusterfsplugin-69jvq 2/2 Running 0 22m
csi-nodeplugin-glusterfsplugin-bf5kx 2/2 Running 0 22m
csi-nodeplugin-glusterfsplugin-n7wzw 2/2 Running 2 22m
csi-provisioner-glusterfsplugin-0 3/3 Running 0 22m
etcd-9dnxxnvhq2 1/1 Running 0 28m
etcd-9wz7txxx59 1/1 Running 0 26m
etcd-dv6q2n7c27 0/1 Init:0/1 0 26s
etcd-operator-7cb5bd459b-mczrk 1/1 Running 0 28m
gluster-kube1-0 1/1 Running 1 26m
gluster-kube2-0 1/1 Running 3 26m
gluster-kube3-0 1/1 Running 1 26m
[vagrant@kube1 ~]$
- The cluster state is in healthy state.
[vagrant@kube1 ~]$ kubectl get pods -n gcs
NAME READY STATUS RESTARTS AGE
csi-attacher-glusterfsplugin-0 2/2 Running 0 22m
csi-nodeplugin-glusterfsplugin-69jvq 2/2 Running 0 22m
csi-nodeplugin-glusterfsplugin-bf5kx 2/2 Running 0 22m
csi-nodeplugin-glusterfsplugin-n7wzw 2/2 Running 2 22m
csi-provisioner-glusterfsplugin-0 3/3 Running 0 22m
etcd-9dnxxnvhq2 1/1 Running 0 28m
etcd-9wz7txxx59 1/1 Running 0 27m
etcd-dv6q2n7c27 1/1 Running 0 54s
etcd-operator-7cb5bd459b-mczrk 1/1 Running 0 29m
gluster-kube1-0 1/1 Running 1 27m
gluster-kube2-0 1/1 Running 3 27m
gluster-kube3-0 1/1 Running 1 27m
[vagrant@kube1 ~]$
[vagrant@kube1 ~]$
[vagrant@kube1 ~]$
[vagrant@kube1 ~]$
[vagrant@kube1 ~]$
[vagrant@kube1 ~]$ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-d92e016d-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc0 glusterfs-csi 11m
pvc-d959ddbe-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc1 glusterfs-csi 11m
pvc-d9860619-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc2 glusterfs-csi 11m
pvc-d9b5e137-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc3 glusterfs-csi 11m
pvc-d9ec5839-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc4 glusterfs-csi 80s
pvc-da1ac34c-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc5 glusterfs-csi 11m
pvc-da510fc2-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc6 glusterfs-csi 11m
pvc-da95c9c6-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc7 glusterfs-csi 76s
pvc-db002280-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc9 glusterfs-csi 11m
pvc-db342a77-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc10 glusterfs-csi 11m
pvc-db66d2fe-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc11 glusterfs-csi 11m
pvc-db9bcff5-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc12 glusterfs-csi 11m
pvc-dbce239d-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc13 glusterfs-csi 11m
pvc-dc04ef7a-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc14 glusterfs-csi 11m
pvc-dc39d4bc-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc15 glusterfs-csi 11m
pvc-dc6dc420-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc16 glusterfs-csi 64s
pvc-dca4b450-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc17 glusterfs-csi 11m
pvc-dcddb56f-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc18 glusterfs-csi 11m
pvc-dd136269-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc19 glusterfs-csi 11m
pvc-dd459ffd-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc20 glusterfs-csi 11m
pvc-dd7b42c5-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc21 glusterfs-csi 77s
pvc-ddaed68a-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc22 glusterfs-csi 11m
pvc-dde6b8bb-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc23 glusterfs-csi 11m
pvc-de1a6f0f-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc24 glusterfs-csi 80s
pvc-de4d84fc-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc25 glusterfs-csi 11m
pvc-de8758b5-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc26 glusterfs-csi 11m
pvc-dec2600c-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc27 glusterfs-csi 11m
pvc-def3a474-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc28 glusterfs-csi 11m
pvc-df2e25d9-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc29 glusterfs-csi 11m
pvc-df64fa77-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc30 glusterfs-csi 11m
pvc-df99124a-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc31 glusterfs-csi 11m
pvc-dfce5481-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc32 glusterfs-csi 11m
pvc-e0066f0b-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc33 glusterfs-csi 11m
pvc-e0431efb-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc34 glusterfs-csi 11m
pvc-e0925da6-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc35 glusterfs-csi 11m
pvc-e0c314f3-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc36 glusterfs-csi 11m
pvc-e0f896db-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc37 glusterfs-csi 11m
pvc-e12be8b5-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc38 glusterfs-csi 11m
pvc-e1610f73-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc39 glusterfs-csi 85s
pvc-e1cecca8-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc41 glusterfs-csi 85s
pvc-e2072dd3-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc42 glusterfs-csi 84s
pvc-e23c35fa-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc43 glusterfs-csi 82s
pvc-e2764934-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc44 glusterfs-csi 79s
pvc-e2b1a377-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc45 glusterfs-csi 82s
pvc-e2eefa00-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc46 glusterfs-csi 81s
pvc-e326ff76-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc47 glusterfs-csi 80s
pvc-e35b14fa-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc48 glusterfs-csi 78s
pvc-e39289e9-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc49 glusterfs-csi 78s
rhsqa-virt05.lab.eng.blr.redhat.com:/root/isse95_gd2.log glusterd2 logs placed at above location