Storage class parameter update stopped working with k8s 1.27 on GKE

[pebrc]

We need to find an alternative way of attribution. Or make sure none of our environments use the built-in storage classes anymore.

storageclass.storage.k8s.io "premium-rwo" deleted
Error from server (AlreadyExists): storageclasses.storage.k8s.io "premium-rwo" already exists
storageclass.storage.k8s.io "premium-rwo" deleted
error: timed out waiting for the condition
storageclass.storage.k8s.io "premium-rwo" deleted
storageclass.storage.k8s.io/premium-rwo replaced
storageclass.storage.k8s.io "standard-rwo" deleted
Error from server (AlreadyExists): storageclasses.storage.k8s.io "standard-rwo" already exists
storageclass.storage.k8s.io "standard-rwo" deleted
Error from server (AlreadyExists): storageclasses.storage.k8s.io "standard-rwo" already exists
storageclass.storage.k8s.io "standard-rwo" deleted
Error from server (AlreadyExists): storageclasses.storage.k8s.io "standard-rwo" already exists
storageclass.storage.k8s.io "standard-rwo" deleted
error: timed out waiting for the condition
storageclass.storage.k8s.io "standard-rwo" deleted
Error from server (AlreadyExists): storageclasses.storage.k8s.io "standard-rwo" already exists

Looks like our storage class labeling does not work anymore in GKE 1.27

Originally posted by @pebrc in https://github.com/elastic/cloud-on-k8s/issues/7743#issuecomment-2075099145

[pebrc]

Idea: install kyverno in all dev/CI environments (should be mostly already the case), add a policy that prevents us from using the built-in GKE storage classes. Create our own storage classes. We already have e2e-default but duplicate the network attached built-ins from GKE and use those with correct attribution and labelling.