[grafana] PersistentVolumeClaim "grafana" is invalid: spec: Forbidden: spec is immutable after creation

[lknite]

When performing a clean installation I end up with a successful installation that I can use but always get this error that keeps trying to resolve:

error when replacing "/dev/shm/737179964": PersistentVolumeClaim "grafana" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims core.PersistentVolumeClaimSpec{ AccessModes: {"ReadWriteMany"}, Selector: nil, Resources: {Requests: {s"storage": {i: {...}, s: "10Gi", Format: "BinarySI"}}}, - VolumeName: "pvc-0554ac94-8558-430e-beea-8fe73499c4f6", + VolumeName: "", StorageClassName: &"cephfs", VolumeMode: &"Filesystem", ... // 2 identical fields }

My values.yaml file looks like this:

grafana:

  <ingress & grafana.ini removed for clarity>

  persistence:
    enabled: true
    accessModes:
    - ReadWriteMany
    storageClassName: cephfs

Screenshot:

Additional: I can delete the pvc, it is recreated, then within a couple minutes the error comes back.

---

Kubernetes v1.25.3 Helm v3.10.3+g835b733

• name: grafana version: 6.56.1 repository: https://grafana.github.io/helm-charts

[zzvara]

Same here using kube-prometheus-stack 45.28.1.

[prasoon-pxc]

same for me also, I am trying to add my custom volumeName field in prometheus-grafana PVCs but it giving error on Argocd syncing , it would be good if we have optionally provide volumeName field in PVC

[rek]

Exact same issue here when using:

        grafana:
          persistence:
            enabled: true
            type: pvc
            storageClassName: default
            accessModes:
              - ReadWriteOnce
            size: 4Gi
            finalizers:
              - kubernetes.io/pvc-protection

Did we find a way around this?

[DannyAlas]

I had a similar issue when trying to adjust the size. I was able to get around it by manually deleting the PVC/statefulsets (ensure to backup if you need the data!) and redeploying with the new value.

See this StackOverflow for help deleting volumes.

kube-prometheus-stack:58.1.2

[JBOClara]

I encountered the same issue. Even when persistence.lookupVolumeName is set to true. https://github.com/grafana/helm-charts/commit/2e998f40701ea8e401479004755a35f6720fbdea

helm diff upgrade kube-prometheus prometheus-community/kube-prometheus-stack --namespace monitoring -f ./values/prometheus/${ENV}/prometheus-operator-values.yaml --version 62.4.0 --set nameOverride=kube-prometheus  --set grafana.fullnameOverride=kube-prometheus-grafana --debug
Enabled three way merge via the envvar
Executing /Users/<USER>/.asdf/installs/helm/3.15.4/bin/helm version
Executing /Users/<USER>/.asdf/installs/helm/3.15.4/bin/helm get manifest kube-prometheus --namespace monitoring
Executing /Users/<USER>/.asdf/installs/helm/3.15.4/bin/helm version
Executing /Users/<USER>/.asdf/installs/helm/3.15.4/bin/helm template kube-prometheus prometheus-community/kube-prometheus-stack --version 62.4.0 --namespace monitoring --set nameOverride=kube-prometheus --set grafana.fullnameOverride=kube-prometheus-grafana --values secrets://./values/prometheus/staging/prometheus-operator-values.yaml --is-upgrade --dry-run=client
Error: unable to generate manifests: cannot patch "kube-prometheus-grafana" with kind PersistentVolumeClaim: PersistentVolumeClaim "kube-prometheus-grafana" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims
  core.PersistentVolumeClaimSpec{
  	AccessModes:      {"ReadWriteOnce"},
  	Selector:         nil,
  	Resources:        {Requests: {s"storage": {i: {...}, s: "6Gi", Format: "BinarySI"}}},
- 	VolumeName:       "pvc-95d11599-0947-11ea-8a2e-0a721f8abcc8",
+ 	VolumeName:       "",
  	StorageClassName: &"gp2",
  	VolumeMode:       &"Filesystem",
  	... // 2 identical fields
  }

EDIT:

I found that helm diff plugin is using --dry-run=client by default.

helm diff upgrade kube-prometheus prometheus-community/kube-prometheus-stack --namespace monitoring -f secrets://./values/prometheus/${ENV}/prometheus-operator-values.yaml --version 62.4.0 --set nameOverride=kube-prometheus  --set grafana.fullnameOverride=kube-prometheus-grafana --dry-run=server