How can we prevent situation if statefulset is stopping the pod before cockroach node decommission is being executed, so the node is showing up as failed instead of as decommissioned.

[sabuhigr]

If manually scale cluster with statefulset, cluster consider that deleted pod(I mean node) is a part of cluster and is drained.

But expected behaviour is cluster should decommission deleted node.

How to reproduce:

• Create cluster with 3 node
• Scale Cluster to 3->4 node with Operator or StatefulSet
• DownScale Cluster with statefulset directly.(Not execute decomission command inside CockroachDB Cluster)

[sabuhigr]

The last situation is seems like that

  id |                         address                         |                       sql_address                       |  build  |         started_at         |         updated_at         | locality | is_available | is_live | gossiped_replicas | is_decommissioning | membership | is_draining
-----+---------------------------------------------------------+---------------------------------------------------------+---------+----------------------------+----------------------------+----------+--------------+---------+-------------------+--------------------+------------+--------------
   4 | NULL                                                    | NULL                                                    | NULL    | NULL                       | 2022-10-05 20:30:32.827817 | NULL     | false        | false   | NULL              | false              | active     | true
   5 | NULL                                                    | NULL                                                    | NULL    | NULL                       | 2022-10-05 20:34:19.750361 | NULL     | false        | false   | NULL              | false              | active     | true
   6 | NULL                                                    | NULL                                                    | NULL    | NULL                       | 2022-10-05 20:39:01.91958  | NULL     | false        | false   | NULL              | false              | active     | true
   7 | NULL                                                    | NULL                                                    | NULL    | NULL                       | 2022-10-05 22:09:24.074329 | NULL     | false        | false   | NULL              | false              | active     | true