Skaffold Deploy - fails to access Helm chart from private GitHub repo

[bouckj]

Skaffold doesn't seem to have any options for providing Authentication credentials to a private GitHub hosted Helm repository. I currently access my Helm repo via a GitHub access token.

For example, if I were running the helm commands manually, I can execute:

helm repo add example --username "<GitUsername>" --password "${GITHUB_ACCESS_TOKEN}" 'https://raw.githubusercontent.com/group/repo/branch/'

helm install -f local-values.yaml example-release repo/chart

When I try to execute skaffold run --tail, I receive a failed to fetch error because the repo cannot be reached. The same error message when I don't provide the auth information when running helm repo add manually. See below for details.

Expected behavior

Skaffold deployment succeeds for accessing and pulling a Helm chart from a private remote repository.

Actual behavior

Skaffold deployment is failing with the following error:

Error: INSTALLATION FAILED: looks like "https://raw.githubusercontent.com/discovery-digital/gauth-ci/github-pages/docs/" is not a valid chart repository or cannot be reached: failed to fetch https://raw.githubusercontent.com/discovery-digital/gauth-ci/github-pages/docs/index.yaml : 404 Not Found

Information

• Skaffold version: v1.37.0

• Operating system: macOS Big Sur 11.6.3

• Installed via: curl -Lo skaffold https://storage.googleapis.com/skaffold/releases/latest/skaffold-linux-amd64 &&
  sudo install skaffold /usr/local/bin/

• Contents of skaffold.yaml:

apiVersion: skaffold/v2beta28
kind: Config
build:
  tagPolicy:
    sha256: {}
  artifacts:
  - image: skaffold-helm
deploy:
  helm:
    releases:
    - name: skaffold-helm-release
      repo: https://raw.githubusercontent.com/group/repo/branch/
      remoteChart: chart-name
      artifactOverrides:
        imageKey: skaffold-helm
      imageStrategy:
        helm: {}
      valuesFiles:
        - local-values.yaml

Steps to reproduce the behavior

1. Create a private GitHub repository
2. Create a branch for serving as a Helm repo
3. Create GitHub access token & set it's value to an environment variable named GITHUB_ACCESS_TOKEN
4. Add a packaged Helm chart to the repo branch
5. Add a Helm index.yaml file to the repo branch (helm repo index .)
6. Open a project locally that you want to build and deploy with Skaffold
7. Add a local-values.yaml file to the root directory as well as the skaffold.yaml above
8. skaffold run --tail

Debug Logs

DEBU[0003] Running command: [helm --kube-context minikube install skaffold-helm-release service --repo https://raw.githubusercontent.com/group/repo/branch/ --set-string imageKey.repository=skaffold-helm,imageKey.tag=xxxxxxxxxxx -f local-values.yaml] subtask=0 task=Deploy Error: INSTALLATION FAILED: looks like "https://raw.githubusercontent.com/group/repo/branch/" is not a valid chart repository or cannot be reached: failed to fetch https://raw.githubusercontent.com/group/repo/branch/index.yaml : 404 Not Found

helm repo add example-repo 'https://raw.githubusercontent.com/group/repo/branch/' Error: looks like "https://raw.githubusercontent.com/group/repo/branch/" is not a valid chart repository or cannot be reached: failed to fetch https://raw.githubusercontent.com/group/repo/branch/index.yaml : 404 Not Found

helm repo add example-repo --username "<GitUsername>" --password "${GITHUB_ACCESS_TOKEN}" 'https://raw.githubusercontent.com/group/repo/branch/' "example-repo" has been added to your repositories

[tejal29]

@bouckj does the command below work?

helm --kube-context minikube install skaffold-helm-release service --repo https://raw.githubusercontent.com/group/repo/branch/ --set-string imageKey.repository=skaffold-helm,imageKey.tag=xxxxxxxxxxx -f local-values.yaml

Maybe skaffold has top pass in the username and password like in helm repo add command

helm repo add example-repo --username "" --password "${GITHUB_ACCESS_TOKEN}" 'https://raw.githubusercontent.com/group/repo/branch/'
``

[bouckj]

I have no problem with helm commands when giving the password. Skaffold doesn't seem to support that field though, or at least it doesn't document or provide examples for it anywhere.

[ondrej-ivanko]

Hi @bouckj,

I found the not very good authentication solution. Company I work for have skaffold.yaml file, which instruct skaffold to deploy via helm and fetch remoteChart of certain version. Unfortunately, when authentication and authorization is necessary to fetch remoteChart, skaffold manges to authenticate to remote helm registry, but you have to hardcode the password in the skaffold.yaml file like so:

apiVersion: skaffold/v2beta15
kind: Config

profiles:
  - name: deploy
    deploy:
      helm:
        releases:
          - name: "{{.SITE}}"
            remoteChart: starship-block
            repo: <url to chart museum>
            version: "{{.VERSION}}"
            namespace: "{{.SITE}}"
            createNamespace: true
            valuesFiles:
              - "siteconfig/{{.SITE}}.yaml"
            upgradeOnChange: true
        flags:
          upgrade:
            - "--username=<USERNAME>"
            - "--password=<PASSWORD>"
          install:
            - "--username=<USERNAME>"
            - "--password=<PASSWORD>"

The inability of skaffold config to interpolate environment variables in authentication credentials flags makes it virtually unusable for remote authentication as most developers would want to keep their passwords safe as possible and not exposing them to deploy pipeline, not mentioning hardcoding them as passwords can change quite often. I would like to see option to write authentication details like this: --password={{.PASSWORD}}

[bouckj]

@ondrej-ivanko Thank you for the example.

Does anyone on this project know of a way to add this private helm repo authentication to Skaffold via a Kubernetes secret, ENV variable, or any other way that doesn't involve hard-coding a password into code?

[ondrej-ivanko]

Hey @bouckj ,

when I was working on this issue I realized that helm add repo <repo name> <repo url> --username --password should work by default and you shouldn't have to log in again for deployment with helm (helm upgrade, helm install). I browsed a little and found this issue 9599. Apparently after removing repo key from skaffold.yaml and just using key-value remoteChart: <repo name>/<chart name> does not require further authentication credentials for helm install/upgrade. I think both those commands are just wrappers for helm pull (because you first need to pull remote chart) and repo key adds --repo flag to it. Than urls of --repo flag and added helm repo are compared and helm somehow assume that they are different under the hood and require further authentication from helm upgrade/install.

I tried it and it works for me. No need for extra flags in skaffold.yaml. Apparently there is pull request fixing this that might end up to be used in helm 3.10.0 version: https://github.com/helm/helm/pull/9760

Hope this helped.

Cheers.

[ericzzzzzzz]

@ondrej-ivanko thank you for providing the solution.

Closing this as specifying chart with remoteChart: / doesn't require further authentication.