Operator "manual" mode (fixed # of nodes)

[JohnStrunk]

Describe the feature you'd like to have. It should be possible to manually specify the number of Gluster nodes and the operator should ensure that number of nodes remains operational.

What is the value to the end user? (why is it a priority?) This capability allows users to manually scale their gluster-based storage to meet their needs.

How will we know we have a good solution? (acceptance criteria)

• By setting operatorMode.mode: manual and specifying operatorMode.manual.nodeCount: n the user can change the size of the gluster cluster
• Increasing nodeCount causes additional nodes to be created and probed into the cluster
• Decreasing nodeCount causes one or more nodes to be drained of bricks, removed from the cluster (detach), and have the pod + backing PV deleted.
• Deletion of a node waits for bricks to be drained before detach and deletion
• Implements node state machine with GD2 to properly synchronize removal

Work items

• [ ] Operator can create initial Gluster node
  • Creation of Gluster PVCs and Deployments
  • Creation of CRs for Gluster Deployments
  • Create service for new Gluster cluster
• [ ] Operator can add nodes to an existing cluster
  • Probe new node into existing cluster
• [ ] Operator can shrink cluster
• [ ] Operator can set/read node state variables
  • Sets "failed" state when desire removal
  • Waits for "abandoned" before deleting node

Additional context Depends on GD2 to drain the nodes of bricks based on node state tag