Michael Burman

Reopen if revisit this later.

The size does not seem to be the issue in other ways than Kubernetes trying to update the Spec. So, from what I can see in the logs is that...

Here's the relevant part from my debug logs: ``` 2023-11-13T12:41:08.259Z INFO api Validating webhook called for update 2023-11-13T12:41:08.259Z INFO api differences in storageConfigs: &PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{1099511627776 0} {} BinarySI},},Claims:[]ResourceClaim{},},VolumeName:,Selector:nil,StorageClassName:*standard,VolumeMode:nil,DataSource:nil,DataSourceRef:nil,} vs &PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage:...

Hey, that's sadly a limitation of StatefulSets (and some StorageClasses), see https://github.com/k8ssandra/cass-operator/issues/263 for more details.

Sadly there I think the Gi is the correct method.. ``` ➜ cass-operator git:(master) ✗ kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE server-data-cluster2-dc2-r1-sts-0 Bound pvc-9d5f342a-1898-42cb-8b0f-748aa9d554a8 1536Gi...

There's no such thing, but that seems you really did try to change the serviceAccount of CassandraDatacenter object (I guess from the K8ssandraCluster object). That error comes from a webhook,...

Quickly looking, the metric you want to drop is: `org_apache_cassandra_metrics_table_view_lock_acquire_time`, the `_bucket` postfix is an indication to Prometheus that this is a Histogram.

Would you happen to have more log lines? It's not really that k8ssandra-operator wouldn't delete seeds, but it looks like the underlying platform (controller-runtime) is having issues when it can't...

I'll try to address some points in here. First of all, we don't encourage using PodTemplateSpec, as there's a lot of things - as you noticed - that could break...

We're rewriting the /metrics endpoint, which should solve these issues: https://github.com/k8ssandra/management-api-for-apache-cassandra/pull/238