prometheus-community
[kube- prometheus-stack] Grafana persistant error
Describe the bug a clear and concise description of what the bug is.
Hi I'm getting error when trying to update kube-prometheus-stack from custom values.yaml.
I'm getting this error when enabled grafana persistant.
Want help.
Upgrading release=kube-stack, chart=prometheus-community/kube-prometheus-stack
FAILED RELEASES: NAME CHART VERSION kube-stack prometheus-community/kube-prometheus-stack
in ./helmfile-main.yaml: failed processing release kube-stack: command "/usr/bin/helm" exited with non-zero status:
PATH: /usr/bin/helm
ARGS: 0: helm (4 bytes) 1: upgrade (7 bytes) 2: --install (9 bytes) 3: kube-stack (10 bytes) 4: prometheus-community/kube-prometheus-stack (42 bytes) 5: --version (9 bytes) 6: 47.4.0 (6 bytes) 7: --force (7 bytes) 8: --cleanup-on-fail (17 bytes) 9: --create-namespace (18 bytes) 10: --kube-context (14 bytes) 11: platform-prod-eks (17 bytes) 12: --namespace (11 bytes) 13: monitoring (10 bytes) 14: --values (8 bytes) 15: /tmp/helmfile2381143591/monitoring-kube-stack-values-59b9f799c4 (63 bytes) 16: --values (8 bytes) 17: /tmp/helmfile4231529747/monitoring-kube-stack-values-789d4cd49f (63 bytes) 18: --reset-values (14 bytes) 19: --history-max (13 bytes) 20: 10 (2 bytes)
ERROR: exit status 1
EXIT STATUS 1
STDERR: W0710 15:01:42.126918 55397 warnings.go:70] unknown field "spec.hostNetwork" W0710 15:01:42.126943 55397 warnings.go:70] unknown field "spec.scrapeConfigNamespaceSelector" W0710 15:01:42.126949 55397 warnings.go:70] unknown field "spec.scrapeConfigSelector" W0710 15:01:42.126953 55397 warnings.go:70] unknown field "spec.tsdb" W0710 15:01:56.255878 55397 warnings.go:70] unknown field "spec.attachMetadata" W0710 15:01:56.852668 55397 warnings.go:70] unknown field "spec.endpoints[0].enableHttp2" Error: UPGRADE FAILED: failed to replace object: PersistentVolumeClaim "kube-stack-grafana" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims core.PersistentVolumeClaimSpec{ AccessModes: {"ReadWriteMany"}, Selector: nil, Resources: {Requests: {s"storage": {i: {...}, s: "10Gi", Format: "BinarySI"}}}, - VolumeName: "pvc-cc05e917-6832-4efa-a995-144ca9ef5c90", + VolumeName: "", StorageClassName: &"efs-sc", VolumeMode: &"Filesystem", ... // 2 identical fields }
COMBINED OUTPUT: W0710 15:01:42.126918 55397 warnings.go:70] unknown field "spec.hostNetwork" W0710 15:01:42.126943 55397 warnings.go:70] unknown field "spec.scrapeConfigNamespaceSelector" W0710 15:01:42.126949 55397 warnings.go:70] unknown field "spec.scrapeConfigSelector" W0710 15:01:42.126953 55397 warnings.go:70] unknown field "spec.tsdb" W0710 15:01:56.255878 55397 warnings.go:70] unknown field "spec.attachMetadata" W0710 15:01:56.852668 55397 warnings.go:70] unknown field "spec.endpoints[0].enableHttp2" Error: UPGRADE FAILED: failed to replace object: PersistentVolumeClaim "kube-stack-grafana" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims core.PersistentVolumeClaimSpec{ AccessModes: {"ReadWriteMany"}, Selector: nil, Resources: {Requests: {s"storage": {i: {...}, s: "10Gi", Format: "BinarySI"}}}, - VolumeName: "pvc-cc05e917-6832-4efa-a995-144ca9ef5c90", + VolumeName: "", StorageClassName: &"efs-sc", VolumeMode: &"Filesystem", ... // 2 identical fields }
What's your helm version?
Version:"v3.12.0"
What's your kubectl version?
v1.27.3
Which chart?
prometheus-community/kube-prometheus-stack
What's the chart version?
47.4.0
What happened?
Hi I'm getting error when trying to update kube-prometheus-stack from custom values.yaml.
I'm getting this error when enabled grafana persistant.
Want help.
Upgrading release=kube-stack, chart=prometheus-community/kube-prometheus-stack
FAILED RELEASES: NAME CHART VERSION kube-stack prometheus-community/kube-prometheus-stack
in ./helmfile-main.yaml: failed processing release kube-stack: command "/usr/bin/helm" exited with non-zero status:
PATH: /usr/bin/helm
ARGS: 0: helm (4 bytes) 1: upgrade (7 bytes) 2: --install (9 bytes) 3: kube-stack (10 bytes) 4: prometheus-community/kube-prometheus-stack (42 bytes) 5: --version (9 bytes) 6: 47.4.0 (6 bytes) 7: --force (7 bytes) 8: --cleanup-on-fail (17 bytes) 9: --create-namespace (18 bytes) 10: --kube-context (14 bytes) 11: platform-prod-eks (17 bytes) 12: --namespace (11 bytes) 13: monitoring (10 bytes) 14: --values (8 bytes) 15: /tmp/helmfile2381143591/monitoring-kube-stack-values-59b9f799c4 (63 bytes) 16: --values (8 bytes) 17: /tmp/helmfile4231529747/monitoring-kube-stack-values-789d4cd49f (63 bytes) 18: --reset-values (14 bytes) 19: --history-max (13 bytes) 20: 10 (2 bytes)
ERROR: exit status 1
EXIT STATUS 1
STDERR: W0710 15:01:42.126918 55397 warnings.go:70] unknown field "spec.hostNetwork" W0710 15:01:42.126943 55397 warnings.go:70] unknown field "spec.scrapeConfigNamespaceSelector" W0710 15:01:42.126949 55397 warnings.go:70] unknown field "spec.scrapeConfigSelector" W0710 15:01:42.126953 55397 warnings.go:70] unknown field "spec.tsdb" W0710 15:01:56.255878 55397 warnings.go:70] unknown field "spec.attachMetadata" W0710 15:01:56.852668 55397 warnings.go:70] unknown field "spec.endpoints[0].enableHttp2" Error: UPGRADE FAILED: failed to replace object: PersistentVolumeClaim "kube-stack-grafana" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims core.PersistentVolumeClaimSpec{ AccessModes: {"ReadWriteMany"}, Selector: nil, Resources: {Requests: {s"storage": {i: {...}, s: "10Gi", Format: "BinarySI"}}}, - VolumeName: "pvc-cc05e917-6832-4efa-a995-144ca9ef5c90", + VolumeName: "", StorageClassName: &"efs-sc", VolumeMode: &"Filesystem", ... // 2 identical fields }
COMBINED OUTPUT: W0710 15:01:42.126918 55397 warnings.go:70] unknown field "spec.hostNetwork" W0710 15:01:42.126943 55397 warnings.go:70] unknown field "spec.scrapeConfigNamespaceSelector" W0710 15:01:42.126949 55397 warnings.go:70] unknown field "spec.scrapeConfigSelector" W0710 15:01:42.126953 55397 warnings.go:70] unknown field "spec.tsdb" W0710 15:01:56.255878 55397 warnings.go:70] unknown field "spec.attachMetadata" W0710 15:01:56.852668 55397 warnings.go:70] unknown field "spec.endpoints[0].enableHttp2" Error: UPGRADE FAILED: failed to replace object: PersistentVolumeClaim "kube-stack-grafana" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims core.PersistentVolumeClaimSpec{ AccessModes: {"ReadWriteMany"}, Selector: nil, Resources: {Requests: {s"storage": {i: {...}, s: "10Gi", Format: "BinarySI"}}}, - VolumeName: "pvc-cc05e917-6832-4efa-a995-144ca9ef5c90", + VolumeName: "", StorageClassName: &"efs-sc", VolumeMode: &"Filesystem", ... // 2 identical fields }
What you expected to happen?
No response
How to reproduce it?
No response
Enter the changed values of values.yaml?
No response
Enter the command that you execute and failing/misfunctioning.
helmfile -f helmfile-main.yaml --environment prod apply
Anything else we need to know?
No response
We're encountering the same error, and it doesn't appear that there's a way to fix the PVC name in the chart values. We're using Flux to manage deploying this chart but we see the same error from a manual helm upgrade as well.
Since we're deploying this chart using flux, we worked around this by updating flux HelmRelease to add a postRenderer that applies a patch to the kube-stack-grafana PVC to replace the original values for volumeName & storageClass. That said, we'd still like to see this resolved to unwind this hack.
Any update, got the same issue, only by helm uninstall && helm install can successfully upgrade prior make sure that pvc/pv will not be deleted on helm uninstall , to remain data
Looks like there was an issue in upstream grafana chart that is fixed now. Try updating to a newer version that incorporates it
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.
![stale[bot] avatar](https://avatars.githubusercontent.com/in/1724?v=4 "Published by a pub.dev verified publisher"){kind=small}