Properly clean up when deleting a cluster

[JohnStrunk]

Describe the feature you'd like to have. When a user removes the top-level CR that defines a Gluster cluster, the operator should properly clean up the Gluster resources.

• It should wait until all Gluster PVs for the affected cluster have been deleted
• It should then delete all Gluster pods and external etcd instance
• It should delete the Gluster cluster's "South" PVs that were used for bricks
• It should delete any other automatically created resources such as Secrets and ConfigMaps

What is the value to the end user? (why is it a priority?) An end-user should be able to remove Gluster just as easily as they installed it. With this proposal, removal would be:

• Delete the top-level CR for the cluster(s)
• Delete the operator
• Delete CRDs and RBAC rules

How will we know we have a good solution? (acceptance criteria)

• All resources that are automatically created as a part of deploying/maintaining the cluster should be properly cleaned up when the top-level CR is deleted
• The cluster should be protected from deletion as long as 1 or more PVs still exist

Work items

• [ ] Wait for volume deletion
• [ ] Delete Gluster Deployments
• [ ] Delete etcd
• [ ] Delete South PVs & other automatically created resources

Additional context