terraform-provider-flux
terraform-provider-flux copied to clipboard
fluxcd
Ignore fields user guide
Recently we got an issue where the kubectl provider along with the fluxcd provider end up always performing an update-in-place. The fix is rather easy so I thought a user guide may be useful for people running into this issue.
Signed-off-by: Luis Gabriel [email protected]
@luisgj just so that I understand what is happening. You have a mutating webhook which makes changes to the Flux manifests which causes issues with the kubectl provider state. And what you are doing is ignoring any change to any field under spec. Meaning that if a user would modify for example the deployment the provider would not detect the change.
Flux sets labels on each object after bootstrap, why do we tell people to ignore metadata.annotations instead of metadata.labels?
@phillebaba thanks for the review. The ignore_fields property will ignore changes on the live manifest. But it will still detect changes as expected to the parsed manifests from the data resources. So you wont went confusing updates on every plan even if you did not change the flux manifests, but it will still detect changes if you - for example - bump the flux version or stuff like that.
@stefanprodan did a yaml diff and for my specific case, I saw no new labels being added, just annotations and after testing this, it seems no new labels where being dtected. But glad to edit the example to add labels if you deem it useful.
flux diff will not report the labels as they are builtin. See here what labels are added: https://fluxcd.io/docs/components/kustomize/kustomization/#reconciliation
Hi, I've been wondering about this and trying to reproduce this scenario. In my terraform plans, I do see in-place updates for yaml_incluster. Looking at the corresponding object in the cluster, for example source-controller deployment, I see annotation like:
annotations:
deployment.kubernetes.io/revision: "21"
which sure aren't in the terraform configuration.
For such things, I guess using ignore_fields to ignore the annotations is okay, unless the flux manifest starts adding annotations in the future. I believe overwriting these annotations with terraform is harmless for flux install manifests as we ensure our attributes are present and any other attributes added by K8s itself are automatically added by K8s if removed by terraform. I'm not sure if we need to officially document ways to ignore such drifts or rather document what they mean with proper reasoning behind them.
But I don't think we should be ignoring spec. Differences in spec are important to be tracked and need to be reconciled whenever there's a drift detected.
Hi @darkowlzz What k8s provider are you using? Most likely a cloud provider will have mutating webhooks. I had this problem on EKS.
What k8s provider are you using? Most likely a cloud provider will have mutating webhooks. I had this problem on EKS.
I'm using a self-hosted k0s cluster.
If the whole spec is ignored using the ignore_fields kubectl argument, will this provider actually work at all? When there's a new flux version, wouldn't the spec be ignored and everything would become no-op?
As per https://registry.terraform.io/providers/gavinbunney/kubectl/latest/docs/resources/kubectl_manifest#ignore-manifest-fields, individual fields can be ignored. Can we do with ignoring only the specific fields in spec and annotation that are mutated by the mutating webhook?
@darkowlzz I do not think the provider will work if the whole spec is ignored, which was why i was hesitant in the first place. I would be more interested in knowing what mutation is causing issues for the provider.
As far as I know EKS does not mutate anything applied to the cluster by default? Must be something else.
I has the same inquiry at first. That's why I have already made those tests. The provider does work if you change the manifests like bumping the version or other actions. It will detect changes to the parsed_yaml and yaml_body properties so plan will work normal.
yaml_incluster drift seems to be related to https://github.com/gavinbunney/terraform-provider-kubectl/issues/92 .
@luisgj can you share more about the changes done by the mutating webhook?
I did see the spec being updated when upgrading the flux provider version with spec ignored in kubectl_manifest. Not sure why it doesn't completely ignores the spec, but if that's the case, I'd assume it's not doing what it's supposed to do and ignoring spec may not do anything. Maybe it works for specific fields in the spec, which may be more appropriate for the mutations done by the webhook.