[Enhancement] Allow scaling down of a pods in the middle of the cluster

[scholzj]

Right now because of using StatefulSet, you can always scale down only the last broker. You cannot remove for example second broker out of 3. That does not work in all use cases. For example in the following situation:

In case we have local disk and the machine is slowly dying. I’d like to add anemone node and then delete he 2nd one

To be able to do this, we would need to use pods directly and not through stateful set. So it is a big change. But there might be more value in it in storage handling etc.

[scholzj]

Triaged on 31.3.2022: Should be achieved in the future with podsets. Should stay open until then.

[scholzj]

This has been addressed using KafkaNodePools which (when enabled) allow to scale down any node.