kubernetes-elasticsearch-cluster icon indicating copy to clipboard operation
kubernetes-elasticsearch-cluster copied to clipboard

Published 20 hours ago • verified publisher icon pires

Kubernetes cluster restart

Open cpalbrecht opened this issue 8 years ago • 4 comments

This es cluster deployment on Kubernetes uses statefulsets and persistent volumes. What happend if the Kubernetes cluster is broken and needs to be shutdown? Does the Elasticsearch deployment find their corresponding persistent volumes after a cluster restart?

cpalbrecht avatar Jul 06 '17 07:07 cpalbrecht

For as long as it's the same cluster, and data isn't corrupted, I believe so.

pires avatar Jul 31 '17 10:07 pires

Thats a promise of Kubernetes Statefulsets. Each replica in a Statefulset gets a unique and durable hostname and Persistent Volume. As long as your etcd cluster is not corrupted, your Kubernetes Cluster should boot up and recreate the state of the pods before the Kubernetes shutdown. If your etcd cluster data get corrupted, there is no such guarantee.

hikhvar avatar Sep 11 '17 05:09 hikhvar

I've looked at exporting elasticsearch snapshots to google buckets, but these aren't 100% reliable (I/O errors cause incomplete backup/restore ~10% of the time) Is there some way to use GCE disk snapshots to make the kubernetes cluster + persistent disks fully restorable?

bw2 avatar Feb 06 '18 01:02 bw2

looks like there's gradual progress toward snaphot/restore for PVCs: https://github.com/kubernetes/website/issues/1691

bw2 avatar Mar 20 '18 10:03 bw2