Automatic RAFT storage peers scaledown/scaleup

[gnadaban]

Is your feature request related to a problem? Please describe. Currently if the Vault replicas were provisioned using RAFT internal storage without setting podAntiAffinity it is a difficult and error-prone process to make the change.

The cluster needs to be manually scaled down to 1 node, while the RAFT storage manually resized maintaining quorum. After setting the anti affinities, the replicas need to be scaled up again to the maximum amount of nodes possible considering the configured affinity. For example, if cluster-size of 5 is configured, but only 3 Availability Zones are available across nodes, in which case the size should be capped by the real boundary: the number of AZs. Given that the cluster needs to be resized first, scaling up would be limited by the Kubernetes scheduler.

Describe the solution you'd like When podAntiAffinity changes to set, and the storage type is RAFT the vault-operator should automatically scale down the cluster size to 1, also removing raft storage peers while preserving quorum, then it should scale up the cluster size to the configured Vault CR size.

Due to the potentially catastrophic outcome of such an operation eg. if the node or storage on which the Vault server is shrank to would fail during resizing, it would be great if the single copy holding all RAFT storage data could be backed up if possible, even if Velero backup is not installed. When using S3 backing for eg. automatic unseal key storage, there is already a configured and available location to place such backups.

Describe alternatives you've considered Resizing the RAFT cluster on the same cluster/nodeset has no alternatives currently.

Additional context None.