Allow PVC expansion

[burmanm]

What this PR does: Support modifying the CassandraDataVolumeClaimSpec sizes if the underlying StorageClass supports it and an annotation cassandra.datastax.com/allow-storage-changes: trueis placed on the CassandraDatacenter. The operator will automatically increase the size of the PVC and wait for the expansion to finish.

Which issue(s) this PR fixes: Fixes #263 Fixes #668

Checklist

• [ ] Changes manually tested
• [x] Automated Tests added/updated
• [ ] Documentation added/updated
• [x] CHANGELOG.md updated (not required for documentation PRs)
• [x] CLA Signed: DataStax CLA

[burmanm]

TODO Add new experimental opt-in for this feature (so no one accidentially messes up their cluster). Was thinking cassandra.datastax.com/allow-pvc-expansion. Or prefix with experimental.cassandra.datastax.com to indicate it's not really battle tested yet?

[burmanm]

Those two tests are expected to fail at this point.

[abe-1]

i am trying to use this parameter for apiVersion: k8ssandra.io/v1alpha1, kind: K8ssandraCluster using annotations: cassandra.datastax.com/allow-storage-changes: "true" in the manifest file but seems to be ignored. I am trying to change datacenters.storageConfig.cassandraDataVolumeClaimSpec.resources.requests.storage from 5Gi to 15Gi, but is not working. Does it not work for K8ssandraCluster? Thanks.

[Miles-Garnsey]

@abe-1 I'm pretty sure you'd need to get the annotations onto the cassDc objects, which you can probably do through K8ssandraCluster's Datacenters[index].Meta.Tags.Annotations.