Deleting backups

[brownleej]

Once we are starting backups as part of the cluster lifecycle, we will need a mechanism to clean up backups as part of deleting a cluster. I think that as part of deleting a cluster we should create a job to run the backup deletion command for the cluster's backup. By default it will run immediately, but there should be a configurable delay so that the user can keep the backup around in case the cluster was deleted inadvertently.

[brownleej]

The backup resource is now separate from the cluster resource, so we can decouple the deletion of the backup from the deletion of the cluster more easily. There's still work we can do to make the backup deletion easier. I think we can add a Deleted state in the backup spec, which will cause the operator to delete the backup. We can use the status to track whether the backup has been fully deleted from the remote store. Once it is fully deleted, it will be safe for the user to delete the backup resource.

Deleting backups is a slow, blocking process. I think we should try using a Job resource to handle the deletion work, and track the completion of that job from the operator.

[johscheuer]

I would propose to implement this as a finalizer and once the FDBBackup resource is deleted the deletion logic will be triggered. We should add a new setting like deleteOnRemoval (I'm open for better names) that allows a user to opt-out of the deletion logic, the default would be true.