substituteFrom value used in a yaml template are not being rendered correctly when used with quotes.

[jbehrends]

trafficstars

I believe this the correct project for this bug report, but if it's not I'm more than happy to resubmit it somewhere else.

I have a Kustomization (kustomize.toolkit.fluxcd.io/v1beta2) which pulls values from a configMap using postBuild: substituteFrom: The values are then used in a HelmRelease (helm.toolkit.fluxcd.io/v2beta1) called by this Kustomization. One of the values is an AWS account number which is defined as a string in the configMap. The variable is then used in the HelmRelese manifest and wrapped in quotes. Once the Kustomize controller renders this template and applies it to Kubernetes, the double quotes get stripped away.

Example:

Snippet of my Kustomization file:

---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
...
spec:
  postBuild:
    substituteFrom:
      - kind: ConfigMap
        name: myvalues

Configmap that's being called above:

apiVersion: v1
kind: ConfigMap
metadata:
  name: myvalues
data:
  AWS_ACCOUNT_ID: "887265102384"     # Not our ACTUAL account number. :)
...

Snippet of my Helmrelease yaml file thats called as part of the above Kustomization

---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: myhelmrelease
...
spec:
  values:
      customMetadata:    # Experiment with adding quotes multiple ways.
        - name: aws_account_id1
          value: '887265102384'   # Passing in a static value with quotes as a control
        - name: aws_account_id2
          value: "${AWS_ACCOUNT_ID}"   # Value wrapped in double quotes
        - name: aws_account_id3
          value: '${AWS_ACCOUNT_ID}'   # Value wrapped in single quotes   
        - name: aws_account_id4
          value: "'${AWS_ACCOUNT_ID}'"   # Value wrapped in double then single quotes
        - name: aws_account_id5
          value: '"${AWS_ACCOUNT_ID}"'   # Value wrapped in single then double quotes
        - name: aws_account_id6
          value: ${AWS_ACCOUNT_ID}       # Value without any quotes     

Snipped of Helmrelease manifest with values that ends up in my cluster:

$ kubectl get helmrelease myhelmrelease -o yaml
...
      customMetadata:
      - name: aws_account_id1
        value: "887265102384"
      - name: aws_account_id2
        value: 887265102384
      - name: aws_account_id3
        value: 887265102384
      - name: aws_account_id4
        value: '''887265102384'''
      - name: aws_account_id5
        value: '"887265102384"'
      - name: aws_account_id6
        value: 887265102384
...

What I'm expecting is the account number value to be rendered with quotes when using the variable name surrounded by quotes. ie '${AWS_ACCOUNT_ID}' should render to "887265102384".

[kingdonb]

This is a topic worth further discussion, but try this:

clusterAccount: "\"${clusterAccount}\""

(This has come up a few times before)

We found this thread at the Bug Scrub meeting just now, we're talking about following up with either a change to the behavior (although we don't want to break backwards compatibility for folks who have already discovered this workaround,) the last time we discussed it, we said we'd like to document it, or fix it so it's not so difficult to discover a way to work around this issue in another way.

Maybe stated a bit clearer: we're not happy with this as a work-around, but it is hopefully serviceable. Can you let us know if this is enough to get you unblocked?

I'm not sure if we have opened an issue to track this more broadly. It seems like it might be a bit between kustomize-controller and Kustomize upstream from SIG-CLI. (It might even be neither of those, and something which can only be fixed in the YAML parser.)

The issue, broadly speaking or at least as I understand it, is that you need these quotes to be present at the point of substitution, and each time we parse the YAML from a string, the parser will be removing quotes from around any items which are already sure to be parsed as strings.

YAML is sure that this is a string:

data:
  variable: "${SUBSTITUTE_ME}"

and so it will become

data:
  variable: ${SUBSTITUTE_ME}

which is fine, for now ... but then, your number type data comes along, and:

data:
  variable: 887265102384

is ultimately how it comes out. That's a number, and YAML is sure of that too (because it doesn't have any quotes around it.)

I'm not sure that's 100% right, but it is at least the gist of the issue, if not of perfect accuracy. So when handling number types for substitution, (the last time we saw this issue it was an AWS account Id as well!) you currently have to use this unfortunate workaround.

[jbehrends]

@kingdonb Thanks for the suggestion!

I tried that and it did not work.

Tried this in my Flux Kustomize manifest: value: "\"${AWS_ACCOUNT_ID}\""

The result that was this in the Flux HelmRelease that was created by the above: value: '"1234567890"'

The result I WANT is: value:"1234567890"

[kingdonb]

Hmm. This worked for at least two people before who came with this issue. One of them brought it to me, in fact.

I'm not sure what else to suggest. The single quotes must be coming from somewhere. Are you sure it's not written like this also in one of your manifests?

There is also the possibility that we revisit this now, and find a better solution that isn't so hacky. I'm very unsurprised that didn't work anymore. It might be that things have changed already (this "fix" always looked brittle and unstable to me) it could also be a difference due to the new server-side apply behaviors, which have changed recently.

(So, then they may also have to change again...)

[jbehrends]

Yea, I'm not doing anything exotic at all here.. I have a Flux Kustomization that's doing a substitution on a Flux HelmRelease yaml and then applying it to my cluster. Then I'm just looking at the substituted value in the Flux HelmRelease (ie kubectl get helmrelease myrelease -o yaml) ... very simple.

For context as well, here are the versions I'm working with: Kubernetes: 1.21.5 kustomize-controller:v0.18.0 helm-controller:v0.13.0 source-controller:v0.18.0

[jonesbusy]

Hi,

Same issue here

My var get substituted from

 - name: FOO
   value: "${BAR}"

to

 - name: FOO
   value: true

I'm trying to obtain (I need to pass a string, not boolean)

 - name: FOO
   value: "true"

If I try

 - name: FOO
   value: "\"${BAR}\""

My HelmRelease gets

 - name: FOO
   value: '"true"'

I didn't find any solution yet.

Best regards,

[kingdonb]

It seems likely that the upgrade to Server-Side Apply broke this workaround.

I'm not sure what else to suggest, if enclosing the typed value in different variations on these quotes is no longer viable as a workaround.

[romankuzmik]

Current workaround is to define variable as: pulsarGcWaitTime: "'50000'" and then in manifest: PULSAR_PREFIX_gcWaitTime: "${pulsarGcWaitTime:='60000'}" it will be rendered as: PULSAR_PREFIX_gcWaitTime: "50000"

would be nice to have a better solution to be able to propagate typed variable (string in this case)

[hiddeco]

I think this would be worth mentioning in one of the cheat sheets on the website.

[ahothan]

is there any progress on finding/agreeing on a permanent fix for this issue? Or could we introduce a more "standard" templating alternative (e.g. go template)?

[vliska]

I have a same/similar problem with postBuild/substitute (not substituteFrom) - (flux 0.39.0):

apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
...
spec:
  postBuild:
    substitute:
      TENANT_ID: '004'

and HelmRelease

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: some-name-${TENANT_ID}
...
spec:
  values:
    tenantId: !!string "${TENANT_ID}"

No matter what I do (tried every workaround mentioned), it is resolved as

values:
  tenantId: 4

... which is not what is acceptable for me (it should be quoted string that will be replaced in resource names)

Any other idea for a workaround?

[kingdonb]

This is not especially helpful for Flux, but there is a system-based (eg. non-workaround) solution in the Weaveworks product (enterprise software warning):

https://docs.gitops.weave.works/docs/gitops-templates/templates/#rendering-templates

Bringing this up because it might be something we can still aim to add to Kustomize Controller.

Then it can be solved generally without suggesting that you need to buy an enterprise product – in the solved version, you could swap out your render type for templating from the default envsubst mode, and even set your own delimiters in case the default ones are not helping in some particularly rough YAML.

I got really very excited when I found this, since I've seen many Flux users struggle with this issue, and I don't have any workaround to offer which I think is reliable anymore, but there's this. The envsubst templating is not terribly well equipped to handle this issue, because of the time that it's applied (which is, (I think?) after the YAML has already been parsed once, so quotes get removed.)

I wouldn't suggest this could be implemented quickly in Flux because I'm not in a position to volunteer to do it, nor have I checked the design to understand if it could be cleanly implemented directly in Flux without major compromises. But as I have worked with the enterprise Weave GitOps product, and explored the template engine, pipelines, and related capabilities, I found this solution quite handy and definitely thought it needed to get a mention here.

[nick4fake]

We have the same issue, but with booleans. Basically any boolean-like value is converted to boolean and breaks schema validation for most objects. Workarounds don't work for us, as sometimes we use value inside strings (so quotes are put there too), and even for simple strings they still convert values to bool

[linkvt]

I just debugged the code a bit and found out, that the variable substitution keeps quotes if you wrap the value inside another pair of quotes (as also shown in a test).

So in case you can influence how the variables are defined below substitute or in the ConfigMap used in substituteFrom you can specify them like this:

spec:
  postBuild:
    substitute:
      prometheus_port: '"9797"' # "'9797'" also worked for me

The root cause really seems to be that flux does convert back and forth between JSON and YAML and looses the quotes around the substituion expression (meaning ${var}), e.g. here: https://github.com/fluxcd/pkg/blob/658f55ae79d28989cc1b03faa41fb236ab64826e/kustomize/kustomize_varsub.go#L61

I compared the resources output of AsYAML() to MustString() and noticed that the quotes were gone around the substitution expression, which explains the behaviour in the initial post. This is IMO the actual problem, but something that happens due to the ambiguity of yamls strings. What happens next is that the replacements are collected and quotes get lost again as key: value and key: "value" are equal in yaml. Then finally envsubst receives all replacements as strings which means it can't know, if a stringified number should be quoted or not. This is why explicitely including the quotes in the string so that they don't get removed by the yaml parsing works, envsubst just replaces the variables with the values it receives.

[vliska]

@linkvt it works as a workaround only for "first" replacement (kustomization with the substituteFrom), but we have e.g. TENANT_ID that is provided to several child kustomizations (TENANT_ID: "${TENANT_ID}", so it breaks in the 2nd step (child kustomization with the same substituteFrom)

[linkvt]

@vliska I didn't test it but have you tried triple quoting the values, e.g. like key: '"\"value\""'? This might keep the quotes for another substitution.

[vliska]

I tried ... keeps them, but breaks it in the first one (where it's also needed) - it's no longer valid TENANT_ID for us :)