[WIP] Allow skipping draining of pods during update/scaledown

[mikkeloscar]

This is a simple proposal for how we could allow skipping of pod draining when during rolling update or scaledown of an ElasticsearchDataSet. It adds a field spec.skipDraining which when set to true will instruct the operator not to attempt to drain a pod before it is terminated. The field defaults to false.

We could also think about enabling this by default in case there are any VolumeClaimTemplates which is usually where you would want to not drain the pod but just terminate it and let the replacement pod remount the data from the PVC.

Fix #70

[njuettner]

Are we able to somehow cover this in our e2e tests?

[otrosien]

we could, maybe by force-fully causing cluster red: replicas=0 and skipDraining would make the primary shard to get lost. (one of the risks of this setup)

[otrosien]

we could even disallow termination if an index has no replicas.

[mikkeloscar]

we could even disallow termination if an index has no replicas.

Can we figure out if an index is stored on the particular node and then check if it's not replicated via the es API?

But it might also be fine if the index is not replicated because if you have it on a disk and this is good enough for you, you should be allowed to terminate the node assuming it will be replaced. Obviously this depends if the termination is because of cluster rotation or because of scaledown.

[otrosien]

Do we assume the user knows what he's doing? For an initial implementation let's just add this caveat to the readme.

[otrosien]

@mikkeloscar I don't think it's worth to add an end-to-end test for this. It's kind of messy because we not only need to provision the nodes but also create an index and add at least one document to it...

[mikkeloscar]

:+1:

[otrosien]

👍