SOPS doesn't work with overlay patches

[apeschel]

It doesn't seem possible to have a kustomization overlay patch encrypted with SOPS. It seems like the patch is applied before SOPS decrypts the secrets, which means SOPS won't be able to decrypt the final merged secret.

Here's an example:

https://github.com/apeschel/flux-testing/tree/main/secrets

Trying to use this on a cluster results in the following error:

my-secrets      False   decryption failed for 'basic-auth': AES decrypt: Error walking tree: Could not decrypt value: Could not decrypt wit
h AES_GCM: cipher: message authentication failed        main/4f9cfd2077e624ab97ac63b79f5a38946e080cf4   False

Here's some details

• SOPS decrypts both basic-auth.yaml and patch.yaml file with no issues locally
• kustomize build is able to render the YAML fine locally, albeit encrypted.
• flux is able to decrypt and deploy basic-auth.yaml fine as long as the patch isn't enabled.

[stefanprodan]

Yes this was the intended design, we should document this limitation.

[apeschel]

@stefanprodan Ah, it should be fine to close this if it's intended. Although, that leaves the question: what is the workflow for adding an overlay with a secret then?

[stefanprodan]

I don't think you can alter secrets encrypted with SOPS, as SOPS creates a checksum of all the key values.

[apeschel]

@stefanprodan What is the recommended workflow for setting cluster specific secrets for the reference multi-cluster examples? They seem to rely on overlay patches to set cluster specific values, but if that's not possible for secrets...

https://github.com/fluxcd/flux2-multi-tenancy https://github.com/fluxcd/flux2-kustomize-helm-example

[stefanprodan]

There is no overlay here for secrets: https://github.com/fluxcd/flux2-multi-tenancy#onboard-tenants-with-private-repositories

[apeschel]

One solution I'm seeing is to not use overlays at all, and just fully render the cluster specific resource that needs secrets, eg rather than having a patch, only add the fully rendered resource to the flux repository. You would probably need to have some kind of template rendering tool for generating the fully rendered resources though, to prevent a lot of copy pasting.

[stefanprodan]

I think you could use $patch: delete or $patch: replace to swap the whole secret manifest.

[apeschel]

There is no overlay here for secrets: https://github.com/fluxcd/flux2-multi-tenancy#onboard-tenants-with-private-repositories

But what if there was a need for a cluster specific secret? Let's say hypothetically kyverno needed a unique license key added to a configmap the kyverno kustomization created for each cluster its deployed to -- how would you do this with the multi-tenancy repository?

[apeschel]

I think you could use $patch: delete or $patch: replace to swap the whole secret manifest.

That makes sense if you can't manage the base resource to be modified -- let me give that a try. You would still need to render the full resource though.

[stefanprodan]

needed a unique license key added to a configmap the kyverno kustomization created for each cluster

Then add it to ./clusters/my-cluster no? It's unique per cluster so why would consider an overlay?

[apeschel]

Let's say the base layer had a ConfigMap that looks like this:

apiVersion: v1
kind: ConfigMap
metadata:
    name: example
    namespace: default
data:
    database_host: example-mariadb
    username: example
    password: 12345

If you wanted to add a license to that via a patch, the patch would look like this:

apiVersion: v1
kind: ConfigMap
metadata:
    name: example
    namespace: default
data:
    license: 123456abcd

However, if these are encrypted with SOPS, flux would require every cluster to have a resource that looks like this:

apiVersion: v1
kind: ConfigMap
metadata:
    name: example
    namespace: default
data:
    license: <Unique License for Cluster>
    database_host: example-mariadb
    username: example
    password: 12345

So rather than having the common values in a shared base, you have to duplicate them for each cluster.

[Alan01252]

I think this does actually work.

I have a base yaml file with no encrypted data in it.

I then patch that with a patechesStrategicMerge with sops encrypted values ( with individual keys encrypted not the entire file ) and kustomize successfully reconciles in the cluster.

Going one step further I have yet another kustomization over the top, and here's where the caveat exists. If I have any encrypted value in the previous overlays I need to make sure those keys are overwritten ( even if it's the same data ) or kustomize cannot decrypt it, but this doesn't stop me patching any other keys.

Working for me using 0.16.1.

[apeschel]

@Alan01252 It might work in the case that you are merging an unencrypted and encrypted block, but it won't work in the case you're merging two encrypted blocks.

[Alan01252]

Yep. In that case I have to overwrite all the encrypted props but unencrypted stuff overlays fine. Ty for clarification

[scott-grimes]

we have a similar problem with sops-encrypted patches throwing errors like decryption failed for xxx: DataWithFormat: AES decrypt: Error walking tree: Could not decrypt value: Input string xxx does not match sops' data format

In our case we are deploying a HelmRelease with a helm chart which does not allow you to reference sensitive information via a secret. The chart requires that you do something like

spec:
  values:
    debug: true
    database:
      user: asdf
      pass: asdf

It's extremely convenient to be able to run a kustomize build and see the entire HelmRelease configuration in one manifest, what we'd like to do is have a app.yaml for the base helmrelease, and an app.enc.yaml with the sensitive portions, and be able to merge them. Something like

# app.yaml
spec.
  values:
    debug: true

# app.enc.yaml (will be sops-encrypted)
spec:
  values:
    database:
      user: asdf
      pass: asdf

# kustomize.yaml
kind: Kustomization
resources:
  - app.yaml
patchesStrategicMerge:
  - app.enc.yaml