scylla-code-samples icon indicating copy to clipboard operation
scylla-code-samples copied to clipboard

[HELM] When pod is replaced (helm upgrade), it is decommissioned and ring is completely broken

Open Vince-Cercury opened this issue 6 years ago • 5 comments

Using HELM to setup Scylla. If I stop my cluster or do an upgrade of the helm chart or the pods are temporarily down, I get errors :

ERROR 2018-07-02 00:05:10,045 [shard 0] storage_service - This node was decommissioned and will not rejoin the ring unless override_decommission=true has been set,or all existing data is removed and the node is bootstrapped again
ERROR 2018-07-02 00:05:10,045 [shard 0] seastar - Exiting on unhandled exception: std::runtime_error (This node was decommissioned and will not rejoin the ring unless override_decommission=true has been set,or all existing data is removed and the node is bootstrapped again)

I'm used to Cassandra. If a node is unavailable (Kubernetes might be resizing cluster or any reason), it does not matter. The node can stay DN for as long as it wants. It does not get decommissioned. When the node comes back online and reattach it's volume, everything is back to normal.

What is Scylla's behaviour? is it compatible with Kubernetes?

Vince-Cercury avatar Jul 02 '18 00:07 Vince-Cercury