gluster
consider using Embeded ETCD in glusterd2
Currently, we are using external ETCD in GCS, if kubernetes nodes get rebooted, all the pods will get restarted. due to pod restart, if the ETCD pods in the clusters lose the quorum, ETCD operator will not be able to maintain the ETCD cluster and the ETCD pods won't come up automatically
ETCD operator issue: https://github.com/coreos/etcd-operator/issues/1972
Pods after before node restart
[vagrant@kube1 ~]$ kubectl get po -ngcs
NAME READY STATUS RESTARTS AGE
csi-attacher-glusterfsplugin-0 2/2 Running 0 5m
csi-nodeplugin-glusterfsplugin-6mfkz 2/2 Running 0 4m59s
csi-nodeplugin-glusterfsplugin-9894b 2/2 Running 0 4m59s
csi-nodeplugin-glusterfsplugin-kg47n 2/2 Running 0 4m59s
csi-provisioner-glusterfsplugin-0 2/2 Running 0 4m59s
etcd-operator-7cb5bd459b-cvzpj 1/1 Running 0 13m
etcd-pdrj27zbz6 1/1 Running 0 11m
etcd-qnfdg7m4vm 1/1 Running 0 12m
etcd-skblg6rl8m 1/1 Running 0 11m
gluster-kube1-0 1/1 Running 0 10m
gluster-kube2-0 1/1 Running 0 10m
gluster-kube3-0 1/1 Running 0 10m
Pods status After node reboot
[vagrant@kube1 ~]$ kubectl get po -ngcs
NAME READY STATUS RESTARTS AGE
csi-attacher-glusterfsplugin-0 2/2 Running 2 8m17s
csi-nodeplugin-glusterfsplugin-6mfkz 2/2 Running 2 8m16s
csi-nodeplugin-glusterfsplugin-9894b 2/2 Running 2 8m16s
csi-nodeplugin-glusterfsplugin-kg47n 2/2 Running 2 8m16s
csi-provisioner-glusterfsplugin-0 2/2 Running 2 8m16s
etcd-operator-7cb5bd459b-cvzpj 1/1 Running 1 16m
etcd-pdrj27zbz6 0/1 Error 0 14m
etcd-qnfdg7m4vm 0/1 Error 0 15m
etcd-skblg6rl8m 0/1 Error 0 15m
gluster-kube1-0 1/1 Running 1 14m
gluster-kube2-0 1/1 Running 1 14m
gluster-kube3-0 1/1 Running 1 14m
Logs from ETCD operator
time="2018-11-05T05:20:45Z" level=warning msg="all etcd pods are dead." cluster-name=etcd pkg=cluster
time="2018-11-05T05:20:53Z" level=warning msg="all etcd pods are dead." cluster-name=etcd pkg=cluster
time="2018-11-05T05:21:01Z" level=warning msg="all etcd pods are dead." cluster-name=etcd pkg=cluster
time="2018-11-05T05:21:09Z" level=warning msg="all etcd pods are dead." cluster-name=etcd pkg=cluster
time="2018-11-05T05:21:17Z" level=warning msg="all etcd pods are dead." cluster-name=etcd pkg=cluster
time="2018-11-05T05:21:25Z" level=warning msg="all etcd pods are dead." cluster-name=etcd pkg=cluster
time="2018-11-05T05:21:33Z" level=warning msg="all etcd pods are dead." cluster-name=etcd pkg=cluster
time="2018-11-05T05:21:41Z" level=warning msg="all etcd pods are dead." cluster-name=etcd pkg=cluster
time="2018-11-05T05:21:49Z" level=warning msg="all etcd pods are dead." cluster-name=etcd pkg=cluster
time="2018-11-05T05:21:57Z" level=warning msg="all etcd pods are dead." cluster-name=etcd pkg=cluster
time="2018-11-05T05:22:05Z" level=warning msg="all etcd pods are dead." cluster-name=etcd pkg=cluster
time="2018-11-05T05:22:13Z" level=warning msg="all etcd pods are dead." cluster-name=etcd pkg=cluster
time="2018-11-05T05:22:21Z" level=warning msg="all etcd pods are dead." cluster-name=etcd pkg=cluster
time="2018-11-05T05:22:29Z" level=warning msg="all etcd pods are dead." cluster-name=etcd pkg=cluster
time="2018-11-05T05:22:37Z" level=warning msg="all etcd pods are dead." cluster-name=etcd pkg=cluster
time="2018-11-05T05:22:45Z" level=warning msg="all etcd pods are dead." cluster-name=etcd pkg=cluster
time="2018-11-05T05:22:53Z" level=warning msg="all etcd pods are dead." cluster-name=etcd pkg=cluster
time="2018-11-05T05:23:01Z" level=warning msg="all etcd pods are dead." cluster-name=etcd pkg=cluster
time="2018-11-05T05:23:09Z" level=warning msg="all etcd pods are dead." cluster-name=etcd pkg=cluster
Lets make sure to document how we can solve the quorum issue with embedded ETCD, and then proceed with it (if at all).
@Madhu-1 Could you also elaborate what exactly you meant by embedded etcd from the context of GCS deployment, where would the etcd cluster will run and how? I believe this is not the same as running embedded etcd by GD2 itself, correct?
@atinmu I am proposing running embedded ETCD by GD2 itself. currently, we have some issue with using embedded ETCD started by GD2 if we fix those issues we can use it.
Also related is the operator work on adding persistence: https://github.com/coreos/etcd-operator/issues/1323. I'm trying to figure out the status.
WRT using embedded... I don't know that this is going to make it any easier. While we will be able to implement persistence, we will need to manage membership ourselves as gd2 pods come and go. I think that is a problem on the order of difficulty of implementing the etcd operator.
