Allow optional disabling of pvc deletion

[ericsmalling]

What is missing?

Add a boolean toggle to disable the automatic deletion of PVCs by the reconciler during datacenter deletion process.

Why is this needed?

My clients manually manage PVC reclamation in their clusters and if one of their end-users deletes a datacenter, they want to be able to retain the pvc or delete in on their own schedule.

┆Issue is synchronized with this Jira Story by Unito ┆Issue Number: CASS-6

[burmanm]

I'm not a fan of creating new boolean for this, but I'd be happy to accept a PR if it adds an annotation to the CassandraDatacenter to prevent these. We already have two similar ones, "cassandra.datastax.com/no-cleanup"and "cassandra.datastax.com/no-finalizer"

So something like no-pvc-deletion could be added to the CassandraDatacenter and check the existence of it before executing https://github.com/k8ssandra/cass-operator/blob/master/pkg/reconciliation/reconcile_datacenter.go#L91

[ericsmalling]

The no-pvc-deletion CR option sounds like a good idea.

[ericsmalling]

Returning from a break, will be working on this asap

[ericsmalling]

Thank you!