Providing values from ConfigMap and Secret resources

[hiddeco]

To be able to provide values by other means than in-lining them in the HelmRelease resource. We want to provide users with the option to reference to values in ConfigMap and Secret resources.

Spec changes

diff --git a/docs/spec/v2alpha1/helmreleases.md b/docs/spec/v2alpha1/helmreleases.md
index 5c13e45..086f624 100644
--- a/docs/spec/v2alpha1/helmreleases.md
+++ b/docs/spec/v2alpha1/helmreleases.md
@@ -70,6 +70,10 @@ type HelmReleaseSpec struct {
 	// +optional
 	Uninstall Uninstall `json:"uninstall,omitempty"`
 
+	// ValuesFrom holds references to values in Secrets and ConfigMaps.
+	// +optional
+	ValuesFrom []ValuesReference `json:"valuesFrom,omitempty"`
+
 	// Values holds the values for this Helm release.
 	// +optional
 	Values apiextensionsv1.JSON `json:"values,omitempty"`
@@ -96,6 +100,30 @@ type HelmChartTemplate struct {
 	Interval *metav1.Duration `json:"interval,omitempty"`
 }
 
+// ValuesReference contains enough information to let you locate the
+// values reference at namespace level, and where the (sub)set of
+// values should be merged at.
+type ValuesReference struct {
+	// Kind of the values referent.
+	// +kubebuilder:validation:Enum=Secret,ConfigMap
+	// +required
+	Kind string `json:"kind"`
+
+	// Name of the values referent.
+	// +required
+	Name string `json:"name"`
+
+	// ValuesKey is the key in the referent the values can be found at.
+	// Defaults to 'values.yaml'.
+	// +optional
+	ValuesKey string `json:"valuesKey,omitempty"`
+
+	// TargetPath is the YAML dot notation path the values should be merged at.
+	// Defaults to 'None', which results in the values getting merged at the root.
+	// +optional	
+    TargetPath string `json:"targetPath,omitempty"`
+}
+
 // Install holds the configuration for Helm install actions.
 type Install struct {
 	// Timeout is the time to wait for any individual Kubernetes operation (like Jobs

Changes vs Helm Operator's valuesFrom

• Values from external sources, for example an URL, are no longer supported.

  This choice was made because you can not reliably restore the cluster state from Git repositories if the configuration of a service relies on some URL being available. In addition to this, having the optional param is even more problematic, as the cluster state will change if the URL is briefly unavailable during reconciliation.

• Chart file references are no longer supported, this will instead be part of the HelmChart API. https://github.com/fluxcd/helm-controller/issues/4

• It is possible to define a targetPath where the values should be merged at. This works the same as --set <YAML dot notation path>=<value>. With this, the following Helm action values:

  helm install \
    --set-file global.identityTrustAnchorsPEM=ca.crt \
    --set-file identity.issuer.tls.crtPEM=issuer.crt \
    --set-file identity.issuer.tls.keyPEM=issuer.key \
    --set identity.issuer.crtExpiry=$(date -d '+8760 hour' +"%Y-%m-%dT%H:%M:%SZ") \
    linkerd/linkerd2
  

  Can be created as a secret using kubectl:

  kubectl create secret generic cert-values \
    --from-file=ca.crt=./certs/ca.crt \
    --from-file=issuer.crt=./certs/issuer.crt \
    --from-file=issuer.key=./certs/issuer.key \
    --from-literal=crtExpiry=$(date -d '+8760 hour' +"%Y-%m-%dT%H:%M:%SZ")
  

  And then be declaratively defined in the HelmRelease as:

  spec:
    valuesFrom:
    - kind: Secret
      name: cert-values
      valuesKey: ca.crt
      targetPath: global.identityTrustAnchorsPEM
    - kind: Secret
      name: cert-values
      valuesKey: issuer.crt
      targetPath: identity.issuer.tls.crtPEM
    - kind: Secret
      name: cert-values
      valuesKey: issuer.key
      targetPath: identity.issuer.tls.keyPEM
    - kind: Secret
      name: cert-values
      valuesKey: crtExpiry
      targetPath: identity.issuer.crtExpiry
  

Behavioral notes

• Referenced values should be watched, and changes should trigger a reconciliation of the HelmRelease. This may require some documentation around the fact that this may need some coordination during changes to multiple referenced resources, either by:
  • A pattern of creating new ConfigMap and Secret resources each time using a hash in the name.
  • Combining values in the same file that are likely to receive updates at the same time.

Tasks

• [x] Support referring to ConfigMap and Secret resources
• [x] Support merging at TargetPath level
• [ ] Watch referred resources

Discussions / references

• https://github.com/fluxcd/toolkit/discussions/100

[stealthybox]

This choice was made because you can not reliably restore the cluster state from Git repositories if the configuration of a service relies on some URL being available.

We could document a workaround for migration. A user could create a CronJob in the same namespace that curls a URL into a ConfigMap using a config-editor ServiceAccount.

The HelmRelease may then depend on that.

The CronJob can choose what to do when the URL becomes unavailable which allows the user to decide how the error should be handled separately from whether the valueRef's is optional.

[stealthybox]

having the optional param ...

Do we already support optional vs. required valueRefs?

I don't see it in the current API. I was thinking we could basically take the same optional field from Pod's ConfigMapEnvSource

It should be noted that even if we are no longer supporting URL's, ConfigMap/Secrets's with values may often not come from the same gitRepo or even a gitRepo at all. They may come from another gitRepo, be placed in the cluster manually (example above), or be the result of some other controller (ex: cert-manager).

Should we support explicitly optional valueRefs?

[stealthybox]

optional valueRefs would allow helmReleases to change config based on the live state of the cluster. This could be useful for allowing a users to extend behavior at a later date with their own repo without modifying the HelmRelease object. As long as the user can create/modify the referenced Configs, they can turn on/off features and etc. This works kind of like a UNIX style /config.d/ dir.

I do a similar thing with volumes: I like to modify my CoreDNS config so that adding keys to a separate coredns-corefile-extras ConfigMap ends up appending to the config from other repos. This is done through an optional volumeMount for /etc/corefiles.d/.

[Skarlso]

@stealthybox Hello! I would like to offer my help if I can on getting the watch working on the respective resources?

[XtremeAI]

Hi guys! Not sure where this Watching feature is but when values in ConfigMaps or Secrets change there goes no release upgrades and I have not found anything better rather than do flux delete hr <name> to apply the changes which isn't the desired way of course.

[yebyen]

If you're using valuesFrom with a secret ref or configmap the flux reconcile hr command should pick up the changes, or automatically be picked up on the next reconciling event. @XtremeAI is that not the case?

If so you should provide specific reproduction instructions as you may have found a new bug, that is not the subject of this issue report. The watching functionality is only meant to accelerate this behavior, it should be happening on its own if you are patient or if you manually trigger the HelmRelease reconciling.

Please be very specific as this functionality seems to be working for me and so you may have to describe the particular conditions that led to an edge case which I am not naturally going to find on my own.

(Edit: apologies as I may have this confused with another issue report, some of the details don't seem to match the issue I thought I was commenting on.)

[XtremeAI]

Hi @yebyen . Thanks for response. I went and did a bit more research and reconciliation does go ahead which kicks the helm upgrade of the chart. Therefore the issue I am having that pods that rely on new config maps are not getting restarted are probably related to helm upgrade not the HelmRelease controller... so I will dig in that direction I guess ... (upd) OK, found the root cause: https://helm.sh/docs/howto/charts_tips_and_tricks/#automatically-roll-deployments

[kingdonb]

Awesome

I think the recreate-pods is only supported through rollback spec in Helm Controller. That root cause link you mentioned @XtremeAI is perfect, you can enforce the pods recreation through use of an annotation that changes based on the shasum of the configmap. Then you also get the benefit of only recreating pods when the configmap has actually changed under it!

[nvanheuverzwijn]

Something got me really confused with the optional flag. I though that optional was saying that if the values was not found, either the Secret/ConfigMap object OR the valuesKey in the Secret/ConfigMap, it didn't matter. After some testing, I understand that optional is really only saying that if the Secret or ConfigMap does not exist we don't care but if it does exist, you better have that valuesKey defined in those Secret/ConfigMap otherwise, reconciliation will fail.

That feels counterintuitive. If the valuesFrom is really optionnal, reconcilliation should be successful even if the valuesKey does not exist.

Any though ?

[wyndigo-01]

Before I create a new feature request. helm-operator allowed you to specify the source namespace for configmaps and secrets that were used in the valuesFrom. This was very useful, and I'm wondering if there is a specific reason that functionality was removed?

[hiddeco]

See https://fluxcd.io/flux/migration/helm-operator-migration/#values-from-configmap-and-secret-resources-in-other-namespaces-are-no-longer-supported

[nourspace]

I noticed that the file I use to create a configmap/secret can't contain any commas.

# grafana.ini

# This is an okay comment
server:
  root_url: https://grafana.domain.com/
'auth.google':
  enabled: true
  ...
  # This is a bad comment with, a comma. or something like these []
  client_id: ...
...

kubectl create configmap grafana-config --from-file=grafana.ini

# HelmRelease

spec:
  valuesFrom:
  - kind: ConfigMap
    name: grafana-config
    valuesKey: grafana.ini
    targetPath: grafana.grafana\.ini

Otherwise, I'd get

unable to merge value from key 'grafana.ini' in ConfigMap 'monitoring/grafana-config' into target path grafana.grafana.ini': key map " a comma" has no value

Removing the line with the "bad comment" fixes the error.