add options for decryption-provider & decryption-secret to flux_sync

[haraldatbmw]

Terraform-provider-flux should create a gotk-sync.yaml file with decryption-provider and secret-reference inside. Like the flux cli does - see https://toolkit.fluxcd.io/cmd/flux_create_kustomization/

actual

apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
  name: flux-system
  namespace: flux-system
spec:
  interval: 10m0s
  path: ./clusters/dev
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
  validation: client

expected

apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
  name: flux-system
  namespace: flux-system
spec:
  decryption:
    provider: sops
    secretRef:
      name: sops-gpg
  interval: 10m0s
  path: ./clusters/dev
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
  validation: client

[phillebaba]

There is a difference between the create kustomization command and the bootstrap command. The provider is currently limited to the same parameters as the CLI. If you want this feature I think it would be best to first implement it in the CLI. If that is not possible the other method would be to generate a kustomization patch which would modify the generated manifests.

[haraldatbmw]

It think it would be a nice feature to extend the CLI bootstrap command with an encryption option like the command for create a kustomization. If that is a pre-requisite for the terraform provider.

[phillebaba]

So the limitation the provider has is that if the CLI bootstrap command does not support something this provider will not be able to either. So we would need to implement it into the CLI first.

[oxr463]

@phillebaba looks like your PR was merged. But it's still not clear to me how I can customize the gotk-sync.yaml file with the decryption provider options.

[stefanprodan]

@oxr463 you shouldn’t do that, create a different Kustomization with decryption enabled.

[oxr463]

@oxr463 you shouldn’t do that, create a different Kustomization with decryption enabled.

Can you elaborate on why I would need a different Kustomization file?

Here is the current file I have:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- gotk-sync.yaml
- gotk-components.yaml
patchesStrategicMerge:
- patch-notification-controller.yaml
- patch-kustomize-controller.yaml
- patch-source-controller.yaml
- patch-helm-controller.yaml

Source: https://github.com/fluxcd/terraform-provider-flux/blob/83f3f702ef88e55f5006f36ade685cb1efbf42a1/docs/guides/customize-flux-gitlab.md?plain=1#L127

[phillebaba]

I am not really too sure what you mean @oxr463, could you explain this a bit more.

[oxr463]

I just modified it. The link didn't look right.

[oxr463]

See also: https://github.com/fluxcd/flux2/issues/2251

[kingdonb]

you shouldn't [...] customize the gotk-sync.yaml file with the decryption provider options.

This is a generally bad idea because it spoils the capability of Flux's disaster recovery in most use cases, except for a pretty narrow set of use cases (I don't know for sure if it would work, but I'm pretty sure you could get away with this customization if you know what you are doing with Flux's SOPS support and some external KMS provider for decryption).

If your flux-system Kustomization depends on decryption, then in the general case without a KMS provider, a decryption key must be placed in a secret in the flux-system namespace which is created by Flux bootstrap (which presents a chicken-and-egg problem for bootstrap, unless the namespace is already created it can never succeed on a new cluster.)

Someone needs to provide the decryption key ahead of bootstrapping then, and Flux has no mechanism to enforce that for you. So if this customization is not used in flux-system, then Flux can reliably bootstrap itself, and will be able to let you know via events/alerting which of its Kustomizations are failing until the decryption keys have been added.

I'm not sure if that's exactly what was meant when we said you should not do that, but I think that's why we don't recommend it. There are some cases where the secret must be in the same Kustomization and source as some other resources, in general we recommend putting things into one or more additional Kustomizations alongside of the flux-system at the top level, which is reserved for Flux infrastructure and non-tenantized workflows that get cluster admin.

[oxr463]

That's really helpful! Thanks for that summary.