Confusing error message on chart verification error

[andrewnazarov]

I'm facing the following issue. I defined a bunch of external services in my helmfile. Then I tried to sync everything. Doing helm sync I got

Upgrading stable/cert-manager Upgrading gitlab/gitlab-runner Upgrading stable/minio Upgrading stable/prometheus-operator Upgrading stable/chartmuseum Error: failed to download "gitlab/gitlab-runner" (hint: running helm repo update may help)

Error: failed to download "stable/prometheus-operator" (hint: running helm repo update may help)

Error: failed to download "stable/cert-manager" (hint: running helm repo update may help)

Error: failed to download "stable/chartmuseum" (hint: running helm repo update may help)

Error: failed to download "stable/minio" (hint: running helm repo update may help)

err: release "runner" in "helmfile.yaml" failed: exit status 1 err: release "prom" in "helmfile.yaml" failed: exit status 1 err: release "cert-manager" in "helmfile.yaml" failed: exit status 1 err: release "chartmuseum" in "helmfile.yaml" failed: exit status 1 err: release "s3" in "helmfile.yaml" failed: exit status 1 exit status 1

What it's all about? In docs it's stated that helmfile sync updates chart repositories and dependencies under the hood.

Btw, I have another helmfile.yaml located in a different folder, it has a mix of internal (with local charts) and external services. helmfile sync works ok inside this folder.

helmfile version v0.43.2

[mumoshu]

Hey! Thanks for the report.

Would you mind rerunning it with helmfile --log-level=debug and share me the result? Perhaps that helps investigating the underlying issue. 2019年2月6日(水) 20:03 Andrew Nazarov [email protected]:

I'm facing the following issue. I defined a bunch of external services in my helmfile. Then I tried to sync everything. Doing helm sync I got

Upgrading stable/cert-manager Upgrading gitlab/gitlab-runner Upgrading stable/minio Upgrading stable/prometheus-operator Upgrading stable/chartmuseum Error: failed to download "gitlab/gitlab-runner" (hint: running helm repo update may help)

Error: failed to download "stable/prometheus-operator" (hint: running helm repo update may help)

Error: failed to download "stable/cert-manager" (hint: running helm repo update may help)

Error: failed to download "stable/chartmuseum" (hint: running helm repo update may help)

Error: failed to download "stable/minio" (hint: running helm repo update may help)

err: release "runner" in "helmfile.yaml" failed: exit status 1 err: release "prom" in "helmfile.yaml" failed: exit status 1 err: release "cert-manager" in "helmfile.yaml" failed: exit status 1 err: release "chartmuseum" in "helmfile.yaml" failed: exit status 1 err: release "s3" in "helmfile.yaml" failed: exit status 1 exit status 1

What it's all about? In docs it's stated that helmfile sync updates chart repositories and dependencies under the hood.

Btw, I have another helmfile.yaml located in a different folder, it has a mix of internal (with local charts) and external services. helmfile sync works ok inside this folder.

helmfile version v0.43.2

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/roboll/helmfile/issues/467, or mute the thread https://github.com/notifications/unsubscribe-auth/AABV-b_D_ATloI6l0wsYZVF-jxhWfujhks5vKraOgaJpZM4alABX .

[andrewnazarov]

Sure. Here it is (sensitive info is removed):

processing file "helmfile.yaml" in directory "."
second-pass rendering result of "helmfile.yaml":
 0: repositories:
 1:   - name: gitlab
 2:     url: https://charts.gitlab.io
 3:   - name: stable
 4:     url: https://kubernetes-charts.storage.googleapis.com
 5: 
 6: helmDefaults:           
 7:   tillerNamespace: gitlab-managed-apps
 8:   kubeContext: gke_xxxx-XXXX_europe-west3-a_cluster-xxxx
 9:   verify: true
10:   wait: true
11:   timeout: 600
12:   recreatePods: false
13:   force: false
14: 
15: releases:
16:   - name: runner
17:     namespace: gitlab-managed-apps
18:     chart: gitlab/gitlab-runner
19:     version: ~0.1.45
20:     values:
21:     - "./gitlab-runner/gitlab-runner-values.yml"
22: 
23:   - name: prom
24:     namespace: monitoring
25:     chart: stable/prometheus-operator
26:     version: ~2.1.1
27:     values:
28:     - "./prometheus/values.yml"
29: 
30:   - name: s3
31:     namespace: gitlab-managed-apps
32:     chart: stable/minio
33:     version: ~2.4.1
34:     values:
35:     - "./minio/minio-values.yml"
36: 
37:   - name: cert-manager
38:     namespace: default
39:     chart: stable/cert-manager
40:     version: ~0.5.0
41:     set:
42:       - name: rbac.create
43:         value: false
44: 
45:   - name: chartmuseum
46:     namespace: gitlab-managed-apps
47:     chart: stable/chartmuseum
48:     version: ~1.9.0
49:     set:
50:       - name: env.open.STORAGE
51:         value: local
52:       - name: persistence.enabled
53:         value: true
54:       - name: persistence.accessMode
55:         value: ReadWriteOnce
56:       - name: persistence.size
57:         value: 10Gi

Adding repo gitlab https://charts.gitlab.io
exec: helm repo add gitlab https://charts.gitlab.io
"gitlab" has been added to your repositories

Adding repo stable https://kubernetes-charts.storage.googleapis.com
exec: helm repo add stable https://kubernetes-charts.storage.googleapis.com
"stable" has been added to your repositories

Updating repo
exec: helm repo update
Hang tight while we grab the latest from your chart repositories...
...Skip local chart repository
...Successfully got an update from the "gitlab" chart repository
...Successfully got an update from the "stable" chart repository
Update Complete. ⎈ Happy Helming!⎈ 

worker 1/5 started
worker 5/5 started
worker 3/5 started
worker 2/5 started
worker 4/5 started
worker 4/5 finished
worker 2/5 finished
successfully generated the value file at /Users/andrewnazarov/Repo/xxxx-4/infra/init/prometheus/values.yml. produced:
coreDns:
  enabled: false

kubeDns:
  enabled: true

alertmanager:
  alertmanagerSpec:
    storage:
      volumeClaimTemplate:
        spec:
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 10Gi

prometheus:
  prometheusSpec:
    storage:
      volumeClaimTemplate:
        spec:
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 10Gi

grafana:
  ingress:
    enabled: true
    annotations:
      kubernetes.io/ingress.class: "nginx"
      certmanager.k8s.io/cluster-issuer: letsencrypt-prod
    hosts:
      - grafana.test.itc-engineering.com
    tls:
      - secretName: grafana-myingress-cert
        hosts:
          - grafana.test.itc-engineering.com
  persistence:
    enabled: true
    accessModes: ["ReadWriteOnce"]
    size: 10Gi
successfully generated the value file at /Users/andrewnazarov/Repo/xxxx/infra/init/gitlab-runner/gitlab-runner-values.yml. produced:
# This file only contains subset of available configuration, see https://gitlab.com/charts/gitlab-runner/blob/master/values.yaml


## The GitLab Server URL (with protocol) that want to register the runner against
## ref: https://docs.gitlab.com/runner/commands/README.html#gitlab-runner-register
##
gitlabUrl: https://gitlab.itc-engineering.com

## The Registration Token for adding new Runners to the GitLab Server. This must
## be retreived from your GitLab Instance.
## ref: https://docs.gitlab.com/ce/ci/runners/README.html#creating-and-registering-a-runner
##
runnerRegistrationToken: "XXXXXXXXX"

## Unregister all runners before termination
##
## Updating the runner's chart version or configuration will cause the runner container
## to be terminated and created again. This may cause your Gitlab instance to reference
## non-existant runners. Un-registering the runner before termination mitigates this issue.
## ref: https://docs.gitlab.com/runner/commands/README.html#gitlab-runner-unregister
##
unregisterRunners: false

## Configure the maximum number of concurrent jobs
## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section
##
concurrent: 3

## Defines in seconds how often to check GitLab for a new builds
## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section
##
checkInterval: 15

## Configuration for the Pods that that the runner launches for each new job
##
runners:
  
  ## Run all containers with the privileged flag enabled
  ## This will allow the docker:dind image to run if you need to run Docker
  ## commands. Please read the docs before turning this on:
  ## ref: https://docs.gitlab.com/runner/executors/kubernetes.html#using-docker-dind
  ##
  privileged: true

  ## Distributed runners caching
  ## ref: https://gitlab.com/gitlab-org/gitlab-runner/blob/master/docs/configuration/autoscale.md#distributed-runners-caching
  ##
  ## Create a secret 's3access' containing 'accesskey' & 'secretkey'
  ## ref: https://aws.amazon.com/blogs/security/wheres-my-secret-access-key/
  ##
  ## $ kubectl create secret generic s3access --\
  ##   --from-literal=accesskey="YourAccessKey" \
  ##   --from-literal=secretkey="YourSecretKey"
  ## ref: https://kubernetes.io/docs/concepts/configuration/secret/
  ##
  cache:
    cacheType: s3
    s3ServerAddress: s3-minio:9000
    s3BucketName: cache
    s3CacheInsecure: true
    cacheShared: true

  ## Build Container specific configuration
  ##
  builds: 
    memoryLimit: 10Gi
    cpuRequests: 1000m
    memoryRequests: 8Gi

  ## Service Container specific configuration
  ##
  services: {}
    # cpuLimit: 200m
    # memoryLimit: 256Mi
    # cpuRequests: 100m
    # memoryRequests: 128Mi

  ## Helper Container specific configuration
  ##
  helpers: {}
    # cpuLimit: 200m
    # memoryLimit: 256Mi
    # cpuRequests: 100m
    # memoryRequests: 128Mi
    # image: gitlab/gitlab-runner-helper:x86_64-latest


worker 5/5 finished
successfully generated the value file at /Users/andrewnazarov/Repo/xxxx-4/infra/init/minio/minio-values.yml. produced:
# This file only contains subset of available configuration, see https://github.com/kubernetes/charts/blob/master/stable/minio/values.yaml

## Set default accesskey, secretkey
accessKey: "XXXXXXX"
secretKey: "XXXXXXX"

## Create a bucket after minio install
##
defaultBucket:
  enabled: true
  ## If enabled, must be a string with length > 0
  name: cache
  ## Can be one of none|download|upload|public
  policy: none
  ## Purge if bucket exists already
  purge: false

worker 3/5 finished
worker 1/5 finished
worker 5/5 started
worker 3/5 started
worker 4/5 started
Upgrading stable/cert-manager
Upgrading stable/prometheus-operator
Upgrading stable/chartmuseum
exec: helm upgrade --install --reset-values cert-manager stable/cert-manager --version ~0.5.0 --verify --wait --timeout 600 --namespace default --set rbac.create=false --tiller-namespace=gitlab-managed-apps --kube-context=gke_xxxx-XXXXXX_europe-west3-a_cluster-xxxx
exec: helm upgrade --install --reset-values prom stable/prometheus-operator --version ~2.1.1 --verify --wait --timeout 600 --namespace monitoring --values /var/folders/pl/k4wqlgfj3cjf6b2p7qblq1200000gn/T/values637418716 --tiller-namespace=gitlab-managed-apps --kube-context=gke_xxxx-XXXXXX_europe-west3-a_cluster-xxxx
exec: helm upgrade --install --reset-values chartmuseum stable/chartmuseum --version ~1.9.0 --verify --wait --timeout 600 --namespace gitlab-managed-apps --set env.open.STORAGE=local --set persistence.enabled=true --set persistence.accessMode=ReadWriteOnce --set persistence.size=10Gi --tiller-namespace=gitlab-managed-apps --kube-context=gke_xxxx-XXXXXX_europe-west3-a_cluster-xxxx
worker 1/5 started
worker 2/5 started
Upgrading stable/minio
exec: helm upgrade --install --reset-values s3 stable/minio --version ~2.4.1 --verify --wait --timeout 600 --namespace gitlab-managed-apps --values /var/folders/pl/k4wqlgfj3cjf6b2p7qblq1200000gn/T/values565451374 --tiller-namespace=gitlab-managed-apps --kube-context=gke_xxxx-XXXXXX_europe-west3-a_cluster-xxxx
Upgrading gitlab/gitlab-runner
exec: helm upgrade --install --reset-values runner gitlab/gitlab-runner --version ~0.1.45 --verify --wait --timeout 600 --namespace gitlab-managed-apps --values /var/folders/pl/k4wqlgfj3cjf6b2p7qblq1200000gn/T/values120155275 --tiller-namespace=gitlab-managed-apps --kube-context=gke_xxxx-XXXXXX_europe-west3-a_cluster-xxxx
Error: failed to download "gitlab/gitlab-runner" (hint: running `helm repo update` may help)

worker 2/5 finished
Error: failed to download "stable/cert-manager" (hint: running `helm repo update` may help)

worker 3/5 finished
Error: failed to download "stable/minio" (hint: running `helm repo update` may help)

worker 1/5 finished
Error: failed to download "stable/prometheus-operator" (hint: running `helm repo update` may help)

worker 4/5 finished
Error: failed to download "stable/chartmuseum" (hint: running `helm repo update` may help)

worker 5/5 finished
err: release "runner" in "helmfile.yaml" failed: exit status 1
err: release "cert-manager" in "helmfile.yaml" failed: exit status 1
err: release "s3" in "helmfile.yaml" failed: exit status 1
err: release "prom" in "helmfile.yaml" failed: exit status 1
err: release "chartmuseum" in "helmfile.yaml" failed: exit status 1
exit status 1

[andrewnazarov]

My first assumption is probably it has something to do with the version of one of the releases. For example, the version of cert-manager is pinned not to the most recent.

[andrewnazarov]

My first assumption is probably it has something to do with the version of one of the releases. For example, the version of cert-manager is pinned not to the most recent.

No, I've just checked it out. Versions are not the case.

[andrewnazarov]

I found it. helmDefaults.verify=true causes the issue. I set it to false and helmfile sync started working.

By the way, it seems for some reason -i flag doesn't work with --log-level=debug. I'll double check that.

[mumoshu]

Thanks for your effort!

I found it. helmDefaults.verify=true causes the issue. I set it to false and helmfile sync started working.

Interesting. All it does should be just adding --verify to helm commands called by helmfile. Do you have any idea how it resulted in such errors...?

By the way, it seems for some reason -i flag doesn't work with --log-level=debug. I'll double check that.

How the whole set of flags are ordered in your command? I thought both --log-level and -i flags must precede before sync so that the command looks like helm --log-level=debug -i sync.

[andrewnazarov]

Interesting. All it does should be just adding --verify to helm commands called by helmfile. Do you have any idea how it resulted in such errors...?

Probably it has somethng to do with:

helm upgrade --dry-run --debug --install --reset-values chartmuseum stable/chartmuseum --version ~1.9.0 --verify --wait --timeout 600 --namespace gitlab-managed-apps --set env.open.STORAGE=local --set persistence.enabled=true --set persistence.accessMode=ReadWriteOnce --set persistence.size=10Gi --tiller-namespace=gitlab-managed-apps --kube-context=gke_iris-206888_europe-west3-a_cluster-iris
[debug] Created tunnel using local port: '60206'

[debug] SERVER: "127.0.0.1:60206"

Error: Failed to fetch provenance "https://kubernetes-charts.storage.googleapis.com/chartmuseum-1.9.0.tgz.prov"

In that case the error message Error: failed to download "stable/chartmuseum" (hint: running 'helm repo update' may help) is misleading.

[andrewnazarov]

How the whole set of flags are ordered in your command? I thought both --log-level and -i flags must precede before sync so that the command looks like helm --log-level=debug -i sync

The order of flags is right.

history | grep -E 'helmfile.*sync'
  752  helmfile --log-level=debug -i sync
  753  helmfile --log-level=debug -i sync
  754  helmfile -i sync
  756  helmfile -i sync
  757  helmfile -i sync
  761  helmfile -i sync
  767  helmfile -i sync
  768  helmfile -i sync
  777  helmfile -i sync
  778  helmfile -i sync
  779  helmfile --log-level=debug -i sync

But it seems that helmfile sync doesn't respect -i flag at all, even without --log-level flag. I don't want to create a new issue at the moment as there are too much noise and false -positives coming from me)).

[mumoshu]

@andrewnazarov Thanks a lot for your support!

Agreed that helmfile should somehow notice the user about the verification error.

Regarding -i, yes, sorry about that! We had a dedicated issue NOT to accept -i for unsupported helmfile commands #416.

[Xtigyro]

Yesterday with helmfile v0.130 I started seeing the same problem. Still need to validate it completely whether it's the exact same root cause.

[4c74356b41]

@mumoshu just hit this same error, started yesterday as well, same fix helped.

[mumoshu]

@Xtigyro @4c74356b41 On which charts did you start seeing the error? (I doubt if it has something to do with the deprecation of stable and incubator chart repositories

[4c74356b41]

with stable, yeah, but helm pull stable/grafana works fine, so it looks like you are correct here.

[worldofgeese]

Just noticed the same error myself using the bitnami/nginx-ingress-controller chart and verify=true. This is using the latest helmfile release.

[ryanpeach]

Same for jenkins/jenkins and verify=true

[woodcockjosh]

failing for bitnami/external-dns

[hungnp-3073]

Same issue for eks/aws-load-balancer-controller when verify=true.

[abvijaykumar]

I am getting this issue with almost all valid helm charts, which work fine with helm command, looks like this is a bug. and sometimes it works, sometimes it fails.

[mumoshu]

verify works only on signed charts. Have you really tried it with --verify when it didn't reproduce for you with the vanilla helm? AFAIK, charts are rarely signed.

[abvijaykumar]

Yes I tried verify. Jt still fails. Another behaviours I see is that. Sometimes it goes thru with one chart and fails with another chart. And it's random.

[mumoshu]

@worldofgeese bitnami/nginx-ingress-controller seems not signed:

$ helm repo add bitnami https://charts.bitnami.com/bitnami --force-update
"bitnami" has been added to your repositories
$ helm template my-release bitnami/nginx-ingress-controller --verify
Error: failed to download "bitnami/nginx-ingress-controller"

@ryanpeachjenkins/jenkins seems not signed:

$ helm repo add jenkins https://charts.jenkins.io --force-update
"jenkins" has been added to your repositories
$ helm template my-release jenkins/jenkins --verify
Error: failed to download "jenkins/jenkins"

@hungnp-3073 eks/aws-load-balancer-controller seems not signed:

$ helm repo add eks https://aws.github.io/eks-charts
"eks" has been added to your repositories
$ helm template my-release eks/aws-load-balancer-controller --verify
Error: failed to download "eks/aws-load-balancer-controller"

Everyone, check your chart and let me know which chart doesn't work only with helmfile.

[abvijaykumar]

HI

@worldofgeese bitnami/nginx-ingress-controller seems not signed:

$ helm repo add bitnami https://charts.bitnami.com/bitnami --force-update
"bitnami" has been added to your repositories
$ helm template my-release bitnami/nginx-ingress-controller --verify
Error: failed to download "bitnami/nginx-ingress-controller"

@ryanpeachjenkins/jenkins seems not signed:

$ helm repo add jenkins https://charts.jenkins.io --force-update
"jenkins" has been added to your repositories
$ helm template my-release jenkins/jenkins --verify
Error: failed to download "jenkins/jenkins"

@hungnp-3073 eks/aws-load-balancer-controller seems not signed:

$ helm repo add eks https://aws.github.io/eks-charts
"eks" has been added to your repositories
$ helm template my-release eks/aws-load-balancer-controller --verify
Error: failed to download "eks/aws-load-balancer-controller"

Everyone, check your chart and let me know which chart doesn't work only with helmfile.

How to check if its signed or not. I am using all stable charts..and these errors are random, sometimes they go thru, sometimes they fail.

for example it failed with the following error Error: failed to download "prometheus-community/prometheus-postgres-exporter"

This was working yesterday

[mumoshu]

@abvijaykumar Honestly speaking I've never tried to struggle with --verify for community charts as they are mostly unsigned. But anyway- the easiest way I can think of is to run the helm command with and without the --verify option. If it works only without --verify, it's unsigned.

[abvijaykumar]

@abvijaykumar Honestly speaking I've never tried to struggle with --verify for community charts as they are mostly unsigned. But anyway- the easiest way I can think of is to run the helm command with and without the --verify option. If it works only without --verify, it's unsigned.

Here is what my current yaml looks like. DO u see anything wrong.

repositories:

• name: prometheus-community url: https://prometheus-community.github.io/helm-charts

• name: grafana url: https://grafana.github.io/helm-charts

• name: hashicorp url: https://helm.releases.hashicorp.com

• name: jetstack url: https://charts.jetstack.io

• name: ingress-nginx url: https://kubernetes.github.io/ingress-nginx

• name: argocd url: https://argoproj.github.io/argo-helm

helmDefaults: verify: false wait: true waitForJobs: true createNamespace: true

commonLabels: application-name: bozo-book-library

releases:

• name: prom-stack namespace: monitoring chart: prometheus-community/kube-prometheus-stack values:

  • ./values/postgres-exporter-helm-values.yaml

• name: redis-exporter namespace: monitoring chart: prometheus-community/prometheus-redis-exporter values:

  • ./values/redis-exporter-helm-values.yaml

• name: prometheus-postgres-exporter namespace: monitoring chart: prometheus-community/prometheus-postgres-exporter values:

  • ./values/postgres-exporter-helm-values.yaml

• name: loki namespace: monitoring chart: grafana/loki-stack values:

  • ./values/bozo-loki-stack-values.yaml

• name: vault namespace: vault chart: hashicorp/vault values:

  • ./values/helm-vault-raft-values.yaml

• name: cert-manager namespace: cert-manager chart: jetstack/cert-manager values:

  • ./values/cert-manager-values.yaml

• name: ingress-nginx namespace: ingress-nginx chart: ingress-nginx/ingress-nginx

• name: argocd namespace: argocd chart: argo/argo-cd

[mumoshu]

@abvijaykumar Looks good to me. If it doesn't work, perhaps any of your chart repositories are not working correctly?

[abvijaykumar]

@abvijaykumar Looks good to me. If it doesn't work, perhaps any of your chart repositories are not working correctly?

Its actually random, sometimes some repositories go through, sometimes the same repo fails. Its very unpredictable, so I am not able to code this into my GitOps. Planning to move to Terraform