[Feature Request] Add local Helm Chart templates to repository chart release

[sboschman]

Let me start with describing the issue I am trying to solve.

Many (official) Helm charts allow you to specify annotations on object descriptions (e.g. ingress descriptions). Ingress controllers support f.e. authentication by looking at annotations on the ingress objects. E.g. ingress.kubernetes.io/auth-type and ingress.kubernetes.io/auth-secret. This ingress.kubernetes.io/auth-secret annotation defines the name of the k8s secret containing the username/password for basic auth.

But the charts don't offer a way to create the secret object. If the chart did include a secret template, you could have used the helm secret-plugin to render.

Manually creating the secrets feels bad...

Another option would be to create a helmfile release based on a local chart, having this local chart containing the secret template. Imho it would be a lot cleaner if this secret would be part of the same Helm release as f.e. the ingress object.

This makes me wonder if it is sensible to have a helmfile feature to merge arbitrary local Helm templates into the chart before rendering.

helmfile.yaml:

releases:
- name: myprometheus
  chart: stable/prometheus
  values:
  - values.yaml
  secrets:
  - secret/my-prometheus-secret.yaml
  templates:
  - templates/basic-auth-secret.yaml

values.yaml:

server:
  ingress:
    enabled: true
    annotations:
      ingress.kubernetes.io/auth-type: "basic"
      ingress.kubernetes.io/auth-secret: prometheus-basic-auth-secret

templates/basic-auth-secret.yaml:

apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: prometheus-basic-auth-secret
  labels:
    app: {{ template "prometheus.name" . }}
    chart: {{ .Chart.Name }}-{{ .Chart.Version }}
    heritage: {{ .Release.Service }}
    release: {{ .Release.Name }}
data:
  auth: {{ .Values.myPrometheusSecret | b64enc | quote }}

Feature request #392 describes how to get secrets from different backends/sources (Helm secret plugin / Vault / AWS / KeyHub :heart: ) into helmfile. This feature request is about a way to get those secrets into k8s as Secrets as part to the release they belong to.

Dream mode Being able to add arbitrary Helm templates would allow me to do the following.

Define my own custom resource description containing all the metadata I need to sync a secret from a backend/source to a k8s Secret. A custom k8s controller would then be able to sync the secrets between backend/source and k8s. So changing the secret in the central secret store of truth would take effect without the need to do a deployment (helmfile sync).

And... deployers/releasers don't need access to the secrets themselves, as long as they know the metadata. Only the k8s controller requires access to the secrets.

[mumoshu]

@sboschman Hey! Thanks for the request, and the detailed explanation.

If you don't mind paraphrasing, you want adhoc resources to be injected into the helm release, right?

If so - yes, I felt similar frustration while using another charts, and wondered how I could fully manage all those with helm.

That being said, I was thinking that using the incubator/raw chart would be a potential solution here?

https://github.com/helm/charts/tree/master/incubator/raw

With the chart, instead of:

releases:
- name: myprometheus
  chart: stable/prometheus
  values:
  - values.yaml
  secrets:
  - secret/my-prometheus-secret.yaml
  templates:
  - templates/basic-auth-secret.yaml

You'll write:

releases:
- name: myprometheus-secret
  chart: incubator/raw
  values:
  - resources:
    - apiVersion: v1
      kind: Secret
      type: Opaque
      metadata:
        name: prometheus-basic-auth-secret
      stringData:
        auth: {{ env "MY_PROM_SECRET" }}
- name: myprometheus
  chart: stable/prometheus
  values:
  - values.yaml
  secrets:
  - secret/my-prometheus-secret.yaml

A downside of this approach is that you need to inject the prom secret as an envvar OR maintain not only the secret value but the whole secret object within a secret.yaml to be consumed by helmfile's secrets: field.

For that, I was considering to enhance the incubator/raw chart to accept templates settings, that can be used like:

releases:
- name: myprometheus-secret
  chart: incubator/raw
  values:
  - templates:
    - apiVersion: v1
      kind: Secret
      type: Opaque
      metadata:
        name: prometheus-basic-auth-secret
      stringData:
        auth: {{` {{.Values.myPrometheusSecret}} `}}
  secrets:
  - secret/my-prometheus-secret.yaml
- name: myprometheus
  chart: stable/prometheus
  values:
  - values.yaml

I can send a PR to incubator/raw if that makes sense (or if you could do so I'd greatly appreciate it :)

[mumoshu]

PTAL: https://github.com/helm/charts/pull/12422

[mumoshu]

incubator/raw 0.2.0 has been released :)

[sboschman]

@mumoshu Thanks for your elaborate reply.

Yes, it is basically about injecting adhoc / arbitrary resources into the release of a chart. Merging them into the release itself would be great, but I can live with the incubator/raw idea. That saves a lot of boiler plate chart stuff compared to including a local chart.

I have used envvar (required in helm template) for secrets before, and that is less than ideal for sure. Encrypting entire resource objects is also a pain, as they basically become unreadable to us poor humans.

Being able to template stuff into incubator/raw looks like a nice solution to all this. :+1:

I see you beat me to making changes to incubator/raw, going to test them out asap. Thanks for the work!

[sgandon]

I have used the incubator/raw successfully and that is a nice idea. here is an example

- name: test-docker-watcher
  namespace: test
  chart: incubator/raw
  values:
  - templates:
    - | 
      {{- readFile "docker-image-change-watcher.yaml" | nindent 6 }}

where I have a pod manifest defined in the docker-image-change-watcher.yaml file.

One issue though is that the raw chart creates a configmap resource for ci that is present during the chart install :(.

[sboschman]

Do you get the ci configmap using the templates option or did you see it using the resources option @sgandon ?

I get the following results, using helmfile v0.47.0. Using the templates option seems to overwrite the default with the ci configmap.

Use templates (:+1:):

releases:
- name: test-docker-watcher
  namespace: test
  chart: incubator/raw
  values:
  - templates:
    - |
      {{- readFile "docker-image-change-watcher.yaml" | nindent 6 }}

Fetching incubator/raw
---
# Source: raw/templates/resources.yaml

---apiVersion: v1
kind: Pod
metadata:
  labels:
    app: myapp
    chart: raw-0.2.0
    heritage: Tiller
    release: test-docker-watcher
  name: myapp-pod
spec:
  containers:
  - command:
    - sh
    - -c
    - echo Hello Kubernetes! && sleep 3600
    image: busybox
    name: myapp-container

Use resources (:-1:):

releases:
- name: test-docker-watcher
  namespace: test
  chart: incubator/raw
  values:
  - resources:
    - 
      {{- readFile "docker-image-change-watcher.yaml" | nindent 6 }}

Fetching incubator/raw
---
# Source: raw/templates/resources.yaml

---apiVersion: v1
kind: Pod
metadata:
  labels:
    app: myapp
    chart: raw-0.2.0
    heritage: Tiller
    release: test-docker-watcher
  name: myapp-pod
spec:
  containers:
  - command:
    - sh
    - -c
    - echo Hello Kubernetes! && sleep 3600
    image: busybox
    name: myapp-container

---apiVersion: v1
kind: ConfigMap                            <<< what are you doing here ?
metadata:
  labels:
    app: raw
    chart: raw-0.2.0
    heritage: Tiller
    release: test-docker-watcher
  name: raw

Use resources, and explicitly disable templates (:+1: 1:):

releases:
- name: test-docker-watcher
  namespace: test
  chart: incubator/raw
  values:
  - resources:
    - 
      {{- readFile "docker-image-change-watcher.yaml" | nindent 6 }}
  - templates: []

Fetching incubator/raw
---
# Source: raw/templates/resources.yaml

---apiVersion: v1
kind: Pod
metadata:
  labels:
    app: myapp
    chart: raw-0.2.0
    heritage: Tiller
    release: test-docker-watcher
  name: myapp-pod
spec:
  containers:
  - command:
    - sh
    - -c
    - echo Hello Kubernetes! && sleep 3600
    image: busybox
    name: myapp-container

@mumoshu Can we circumvent ci in some other way (https://github.com/helm/charts/blob/master/incubator/raw/values.yaml#L68) ? Everyone using the resources without the new templates gets the ci configmap I think.

[sgandon]

indeed using the template is fine, I did try with resourcesand this is where I saw the ci-configmap.

[mumoshu]

https://github.com/helm/charts/pull/13633 should fix the ci-configmap issue.

[unacceptable]

I still think that this core ask is a good idea. Consider for a moment a helm chart like Jenkins. It might be desirable to store the CasC in a config map to limit the size of the values file:

https://github.com/helm/charts/blob/75a1020b6cce44cba3df11999b277e5c2c43db84/stable/jenkins/values.yaml#L603-L609

then to apply said config map when changing the chart:

repositories:
  - name: stable
    url: https://kubernetes-charts.storage.googleapis.com

releases:
  - name: '{{ requiredEnv "NAME" }}-{{ requiredEnv "PROJECT" }}-jenkins'
    namespace: '{{ requiredEnv "NAME" }}-{{ requiredEnv "PROJECT" }}'
    labels:
      app: '{{ requiredEnv "NAME" }}-{{ requiredEnv "PROJECT" }}-jenkins-master'
    chart: stable/jenkins
    version: ~1.27.0
    hooks:
    - events: ["prepare"]
      command: "kubectl"
      args: ["apply","-f", '{{ requiredEnv "NAME" }}/{{ requiredEnv "PROJECT" }}/casc-config-map.yaml', "-n", '{{ requiredEnv "NAME" }}-{{ requiredEnv "PROJECT" }}']
    values:
    - values.yaml.gotmpl
    wait: true
    timeout: 3600
    force: true

Where the values file will look something like this:

persistence:
  enabled: true
  storageClass:
  annotations: {}
  accessMode: "ReadWriteOnce"
  size: "200Gi"
  volumes:
    - name: casc-config
      configMap:
        name: jenkins-casc-config
  mounts:
    - name: casc-config
      mountPath: /var/jenkins_home/casc_configs
      readOnly: true

While this solution works, it's not as clean as it could be IMHO, as it's impossible to review diffs in the CasC which can lead to problems if some Junior Dev or Old School admin is trying to do some ClickOps.

Maybe there's a way to copy a template to the chart directory using a hook. If so please lemme know and that would solve my issue with the diffs, and I think it would solve the core of this issue. Again not sure if that last part is possible.

[KarstenSiemer]

I did it like this:

releases:
  - name: alertmanager-basic-auth
    chart: incubator/raw
    values:
    - templates:
      - |
        apiVersion: v1
        kind: Secret
        type: Opaque
        metadata:
          name: basic-auth
          namespace: monitoring
        stringData:
          auth: {{ "{{ .Values.auth }}" }}
    secrets:
      - secrets/alertmanager-basic-auth.yaml

the secret is looking like this:

auth: b1234sadf123asas234fsadf1

[mxmorin]

incubator/raw is now deprecated : https://github.com/helm/charts/tree/master/incubator/raw

[mumoshu]

Hey everyone! I was recently playing around with the idea of "a helmfile-builtin raw-chart-like feature" https://github.com/roboll/helmfile/commit/25baebe2b18cc059389b4bb1fcd334383466f0e2. It turn any local directory containing "*.yaml.gotmpl" into a temporary helm chart by rendering each template with Helmfile's template context/funcs/data. Would it help?

[SeWieland]

I would really like that feature in order to be able to add k8s hooks as resources to the charts (instead of using the helmfile hooks directly).. This would be a really powerful feature IMO 👍

[p3lim]

Any chance this can be considered again, since incubator/raw is now deprecated?

It seems like this request was pretty much turned down due to that workaround, while being limiting in terms of integrating with the chart.

[hlesesne]

SO many charts don't include netpols, yet almost every production deployment we have requires one. The ability to add additional templates at release would be extremely helpful.