Use secrets from within a remote chart

[witten]

Native Helm supports installing a remote chart from a charts server. For example:

$ helm repo update              # Make sure we get the latest list of charts
$ helm install stable/mysql
Released smiling-penguin

This is one of the major benefits of using Helm: You can package up your charts as build artifacts, host them on a charts server, and then install them as needed.

However, with helm-secrets, I haven't found a way to remotely install a chart and use the secrets found within that chart. Here's the helm-wrapper example from the helm-secrets documentation:

AWS_PROFILE=sandbox helm-wrapper upgrade \
  helloworld \
  stable/java-app \
  --install \
  --timeout 600 \
  --wait \
  --kube-context=sandbox \
  --namespace=projectx \
  --set global.app_version=bff8fc4 \
  -f helm_vars/projectx/sandbox/us-east-1/java-app/helloworld/secrets.yaml \
  -f helm_vars/projectx/sandbox/us-east-1/java-app/helloworld/values.yaml \
  -f helm_vars/secrets.yaml \
  -f helm_vars/values.yaml

It appears to assume that the source code of the chart is available and unpacked locally. What I would expect to be able to do is perform a helm-wrapper install or upgrade of a remote chart and not specify local helm_vars files to pull secrets from, but rather implicitly or explicitly pull secrets/values from within the chart being installed.

A work-around for this gap is to first helm fetch and unpack the remote chart in question so that the source is available locally. However, this work-around simply doesn't work with multi-chart solutions like helmfile, in which it helm upgrades multiple charts in one go, without the opportunity to intervene with a helm fetch hack.

So, any thoughts on how to support this remote install use case?

[witten]

An implementation idea for this: In wrapper.sh, if the chart name is not a local directory path (if [ ! -d "$chart" ];), then you you could assume that the chart is remote and try to helm fetch --untar it into a temporary directory before trying to decrypt its secrets files locally. That way, users would be able to helm-wrapper install or upgrade remote charts while still (explicitly) pathing into their contained secrets files.

[witten]

It looks like implementing this may actually be much easier after the separate https://github.com/futuresimple/helm-secrets/pull/60 get merged in. That's because the existing wrapper.sh doesn't do any real parameter parsing, so it's hard to programmatically determine the chart name in order to download it. However, the plugin rewrite in https://github.com/futuresimple/helm-secrets/pull/60 does do some parameter parsing on the helm secrets upgrade code path. And so that could potentially be extended to support helm fetching remote charts.

@mhyllander @szibis

[mhyllander]

I'm not sure what the general view is on this. It's true the remote chart would need to be unpacked locally before any contained secrets files can be decrypted. However, I don't think I would package a chart including secrets and upload it to chart server. Aren't charts on a chart server supposed to be generic? I would probably only put a values.yaml file with default values in the remote chart, and if I need to add secrets I would store the secrets file separately in git.

[witten]

Our use case is as follows: We've got a private package server, and we're actually packaging up various configuration (including helm-secrets encrypted credentials) as Helm charts. (And those charts internally publish their config as Kubernetes Secrets.) This means that our helm-secrets files are included in the charts themselves. The main benefit of an approach like this versus keeping the secrets separately in git is that we don't need a source code checkout to deploy. Instead, we can just use the remote chart on the chart server (effectively a build artifact) and deploy it directly.

[mhyllander]

I see. Well, like you said the chart must be downloaded and unpacked locally (where SOPS and the PGP keys are available) so that the secrets files can be decrypted before installing it. In effect "helm fetch" + "helm secrets install".

I think I would probably prefer to implement this as separate commands in the plugin, e.g. "helm secrets rinstall" and "helm secrets rupdate" ("r" for remote), because they are not pure wrappers for "helm install" and "helm upgrade". There would need to be some extra options to specify the paths to values and secrets files relative to the unpacked chart root, which are different from the arguments to -f/--values (which are relative to the current working directory). It would be a cleaner implementation.

Anyway, I think it's a decision for the maintainers of this project.

[witten]

I think you're right that it might be a cleaner implementation for the code paths to be separate, although it might be a slightly nicer user experience if a single command could work for both local and remote use cases. If that model were to be pursued, -f/--values could be interpreted as: Relative to the current working directory if the chart is local, and relative to the unpacked chart root if the chart is remote. So that might not actually require any extra options.

[mhyllander]

Yes, but the more common use case is to install a remote chart and use a local secrets file, like I described above, so you can't assume that -f/--values always refer to a file relative to an unpacked chart when it's remote. That's why I mean that you need a separate set of options for your use case.

[witten]

Ah, gotcha. Yeah, that makes sense.

[faheem-nadeem]

We have a similar use case as @witten. Would love to see this implemented preferably as a separate of commands as @mhyllander described.

[witten]

@faheem-cliqz Note that if you're using Helmfile in conjunction with helm-secrets, a work-around is to use the new Helmfile hooks feature to helm fetch right before installing a particular chart.

[faheem-nadeem]

@witten I have not been using Helmfile, but after reading a bit on it, I am interested. Would you be so kind to post a hook example with Helmfile. Our current process with fetching, decrypting secrets bundled in a remote chart and installing said chart looks something like this:

$ Helm repo update
$ helm fetch org/example-chart --untar
$ helm secrets dec path/to/secret
$ helm install --namespace example path/to/chart --values path/to/values --values /path/to/decrypted_secret
$ rm untarred_chart

Would love to have this all bundled in to a single command.

[witten]

Here's a partial Helmfile example that would handle all of that:

repositories:
  - name: org
    url: https://your-chart-server.example.org
releases:
  - name: example-chart
    chart: org/example-chart
    hooks:
      - events: ["prepare"]
        command: helm
        args: ["fetch", "org/example-chart", "--untar", "--untardir", "."]
      - events: ["cleanup"]
        command: rm
        args: ["-fr", "example-chart"]
    values:
      - ./example-chart/path/to/values
    secrets:
      - ./example-chart/path/to/secret  # Decrypts transparently.

And then running helmfile sync on that would update repos, download the chart, install the release, etc.

[faheem-nadeem]

Sweet. Just my last question @witten. How are you guys storing the Helmfile. Is it a separate repo with a structure for all projects/environments or using the same helm chart repo? We currently have all our charts, secrets, environment-values in a structure like this, the important bit is the profiles folder.

    kubernetes-charts
    ├── stable
    │   ├── project-x
    │   │   ├── profiles
    │   │   │   ├── production
    │   │   │   │   ├── eu-central-1
    │   │   │   │   │   ├── secrets.yaml
    │   │   │   │   │   ├── values.yaml
    │   │   │   │   │   └── sops.yaml
    │   │   │   │   └── us-east-1
    │   │   │   │       ├── secrets.yaml
    │   │   │   │       ├── values.yaml
    │   │   │   │       └── sops.yaml
    │   │   │   └── dev
    │   │   │       ├── eu-central-1
    │   │   │       ├── secrets.yaml
    │   │   │       ├── values.yaml
    │   │   │       └── sops.yaml
    │   │   ├── templates
    │   │   │   ├── _helpers.tpl
    │   │   │   ├── Notes.txt
    │   │   │   ├── deployment.yaml
    │   │   │   ├── configmap.yaml
    │   │   │   ├── service.yaml
    │   │   │   └── sops.yaml
    │   │   ├── chart.yaml
    │   │   ├── README.md
    │   │   └── values.yaml
    ├── incubator
    │   ├── project-y
    │   │   ├── profiles
    │   │   │   └── dev
    │   │   │       └── us-east-1
    │   │   │           ├── secrets.yaml
    │   │   │           ├── values.yaml
    │   │   │           └── sops.yaml
    │   │   ├── templates
    │   │   │   ├── _helpers.tpl
    │   │   │   ├── Notes.txt
    │   │   │   ├── deployment.yaml
    │   │   │   ├── configmap.yaml
    │   │   │   ├── service.yaml
    │   │   │   └── sops.yaml
    │   │   ├── chart.yaml
    │   │   ├── README.md
    │   │   └── values.yaml
    └── README.rst
    └── CONTRIBUTING.rst
    └── Makefile
    └── Jenkinsfile

One suggestion could be storing helmfiles in a separate root folder and this way it won't be bundled in chart artifacts, or stick them in a separate repository.

[witten]

We currently only have a single Helmfile, and do all the per-environment variation within individual charts. So an individual service may have a chart that has files something like:

helm_vars/

• production
  • values.yaml
  • secrets.yaml
• dev
  • values.yaml
  • secrets.yaml

And then use something like this in the Helmfile to pull in the correct environments' values/secrets from each chart at install time:

    values:
      - example-chart/helm_vars/{{ .Environment.Name }}/values.yaml
    secrets:
      - example-chart/helm_vars/{{ .Environment.Name }}/secrets.yaml

That way, secrets/values can be distributed along with each chart that consumes them.

Also, if you're interested in future mechanisms for distributing Helmfiles themselves, you may want to have a gander at https://github.com/roboll/helmfile/issues/195 or https://github.com/roboll/helmfile/issues/153