prometheus in rancher 2.5: failed to provision volume with StorageClass "local-path": claim.Spec.Selector is not supported

[mamiu]

In Rancher 2.5 (which was released yesterday) the installation of monitoring fails because Prometheus creates a PVC with a selector property. Here's the current state of the PVC (with all irrelevant fields removed):

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    volume.beta.kubernetes.io/storage-provisioner: rancher.io/local-path
    volume.kubernetes.io/selected-node: k3d-demo-server-0
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    app: prometheus
    prometheus: rancher-monitoring-prometheus
    manager: k3s
    operation: Update
  name: prometheus-rancher-monitoring-prometheus-db-prometheus-rancher-monitoring-prometheus-0
  namespace: cattle-monitoring-system
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  selector: {}
  storageClassName: local-path
  volumeMode: Filesystem
status:
  phase: Pending

The log of the local-path-provisioner pod show this error message:

[...] failed to provision volume with StorageClass "local-path": claim.Spec.Selector is not supported [...]

I found that issue #56, but it doesn't resolve this error. And editing the PVC doesn't work because of this restriction: Forbidden: spec is immutable after creation except resources.requests for bound claims.

[hwk42]

I also encountered the same problem when install alluxio in k3s. Warning ProvisioningFailed 2m56s (x11 over 84m) rancher.io/local-path_local-path-provisioner-5d7b5579c7-v44hc_82108aeb-d956-42e2-9c31-7ff185df52ad failed to provision volume with StorageClass "local-path": claim.Spec.Selector is not supported

[mamiu]

Interestingly, I get exactly the same error with longhorn. Doesn't seem to be a local-path-provisioner specific problem. However, I also got this error message on more applications than just Prometheus. As a workaround I simply deleted the following lines from the PVC before deploying:

  selector:
    matchExpressions: []
    matchLabels: {}

[ccittadino-ctic]

I am also seeing this issue when deploying Rancher Monitoring to a k3s cluster. The grafana PVC comes up ok but I get the same error as the OP for prometheus. As @hwk42 stated, we cannot update the PVC directly because it's immutable.

I would like to use the workaround that @mamiu calls out but the values.yml that Rancher Monitoring uses doesn't have the 'selector' field in it at all, suggesting it's some kind of default during the install/upgrade process.

Has anyone figured out a solution/workaround?