[kube-prometheus-stack] Error: INSTALLATION FAILED: create: failed to create: Secret "sh.helm.release.v1.kube-prometheus-stack.v1" is invalid: data: Too long: must have at most 1048576 bytes

[XDavidT]

Describe the bug a clear and concise description of what the bug is.

I tried to use one helm to all kube-prometheus-stack and made a new chart, The Chart.yaml:

apiVersion: v2
name: kube-prometheus-stack
description: A Helm chart for kube-prometheus-stack

type: application

version: 0.0.1

appVersion: "0.0.1"
dependencies:
  - name: kube-prometheus-stack
    version: 38.0.2
    repository: https://artifactory.amat.com/artifactory/prometheus-remote

and values is empty (I tried with my values, but it didn't worked, so cleared to see if the issue is in my values)

What's your helm version?

version.BuildInfo{Version:"v3.9.0", GitCommit:"7ceeda6c585217a19a1131663d8cd1f7d641b2a7", GitTreeState:"clean", GoVersion:"go1.17.5"}

What's your kubectl version?

Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.3", GitCommit:"c92036820499fedefec0f847e2054d824aea6cd1", GitTreeState:"clean", BuildDate:"2021-10-27T18:41:28Z", GoVersion:"go1.16.9", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.2", GitCommit:"f66044f4361b9f1f96f0053dd46cb7dce5e990a8", GitTreeState:"clean", BuildDate:"2022-06-15T14:15:38Z", GoVersion:"go1.18.3", Compiler:"gc", Platform:"linux/amd64"} WARNING: version difference between client (1.22) and server (1.24) exceeds the supported minor version skew of +/-1

Which chart?

kube-prometheus-stack

What's the chart version?

38.0.2

What happened?

Got this error:

Error: INSTALLATION FAILED: create: failed to create: Secret "sh.helm.release.v1.kube-prometheus-stack.v1" is invalid: data: Too long: must have at most 1048576 bytes

What you expected to happen?

No response

How to reproduce it?

Use my chart, with empty values file, run helm dependency update and run helm install kube-prometheus-stack .

Enter the changed values of values.yaml?

No response

Enter the command that you execute and failing/misfunctioning.

helm install kube-prometheus-stack

Anything else we need to know?

I have my custom values file, but didn't use it to check it without and see it's working.

[zahradehghanpur]

I also have the same problem when I import a grafana dashboard as a code in my chart. Is there any solution rather than increasing limit on etcd side?

[dvyas1]

Same issue here as well. Looking for solution. @XDavidT did you find any workaround?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[bjthomas1]

I had this error:

Error: create: failed to create: Secret "sh.helm.release.v1.kube-prometheus-stack.v1" is invalid: data: Too long: must have at most 1048576 bytes

When installing the kube-prometheus-stack Helm Chart (version 39.11.0), with the command:

helm upgrade --install kube-prometheus-stack kube-prometheus-stack/ -f kube-prometheus-stack/vals.yaml -n monitoring

The reason it failed in my case was because I dropped a custom values file called vals.yaml into the root of the unzipped Helm chart (e.g. in ./kube-prometheus-stack/vals.yaml). When moved this down a level (e.g. mv ./kube-prometheus-stack/vals.yaml .) the problem was fixed. I don't know the mechanics of why this occurred or how this fixed it.

[zou-weidong]

cd kube-prometheus-stack-41.3.2
helm upgrade --install -f values.yaml  kube-prometheus-stack -n monitoring  .

I had this error:

Error: UPGRADE FAILED: create: failed to create: Secret "sh.helm.release.v1.kube-prometheus-stack.v2" is invalid: data: Too long: must have at most 1048576 bytes

[iskytek]

Error: create: failed to create: Secret "sh.helm.release.v1.prometheus-stack.v1" is invalid: data: Too long: must have at most 1048576 bytes

the mentioned secret is created by helm and stores the information about release. Since the umbrella chart had a lot of sub-charts , the secret had to store a lot of information which exceeded 1MiB. A solution for this was to remove unnecessary files from the sub-charts so that the secret will parse less files than actual.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[stale[bot]]

This issue is being automatically closed due to inactivity.

[PratapSingh13]

Still no relevant answer to this question. Kindly provide appropriate solution for the same.

[joelrebel]

Incase this helps anyone else - In my case I had to add the .git/ directory into .helmignore to get past this error.

[ramfjord]

@joelrebel thanks! It wasn't my .git/ folder but a large tmp/ folder that was getting packed in!

[dzhiwei]

I also have the same problem when I import a grafana dashboard as a code in my chart. Is there any solution rather than increasing limit on etcd side?

The root cause is that the size limit of configmap in namespace is 1M, kube-prometheus-stack stores all grafana dashboard jsons in K8S ConfigMaps. I removed the un-needed dashboard json and compress the json files(use tool to remove blank and new line) in kube-prometheus-stack/templates/grafana/dashboards-1.14/ folder to work-around the issue. Hope it helps!