fluxcd
Feature request: Flux Kustomization to have a DriftDetection option
In the Helm controller the HelmRelease has an option to specify DriftDetection configuration:
https://fluxcd.io/flux/components/helm/helmreleases/#drift-detection
It would be useful to have a similar configuration option for the Kustomization resources to control whether the kustomize-controller should only detect drifts/changes or should actively fix those.
I know the helm controller and the HelmRelease resource is a bit different form the Kustomization, so it might not be that obvious to take the same approach, but this functionality could be extremely useful in scenarios when the Kustomization is suspended and then turned on again.
may I ask any of the kustomization-controller's maintainers opinion on how this could be implemented if it is possible at all? this functionality is there in other gitops controllers and also supported in one of flux's own controllers, but there is a functionality gap in case of the Kustomization based controller...
The drift feature in helm-controller is very specific to how Helm works, Helm stores a fully copy of all manifests in-cluster in Kubernetes Secrets (this is called Helm storage), the drift detection runs for the storage vs cluster, not for Git vs cluster.
In kustomize-controller we don't duplicate the manifests in etcd like Helm, this would be extremely wasteful, there are no plans to do what Helm does.
When flux is used cross-cluster the connection may be lost temporarily and manual changes may be done on the target resource bypassing flux. After this for a grateful recovery it would be good to have a flag in the Kustomization that warns for drift but does not overwrite it. (Then based on the delta we could decide to update git content or let flux overwrite the drift.)
When flux is used cross-cluster the connection may be lost temporarily and manual changes may be done on the target resource bypassing flux. After this for a grateful recovery it would be good to have a flag in the Kustomization that warns for drift but does not overwrite it. (Then based on the delta we could decide to update git content or let flux overwrite the drift.)
This is not GitOps
We didn't designed the Flux controllers to works like a CLI in an imperative way. If you want to do that, then suspend the Flux Kustomizations and use the Flux CLI. There is a flux diff kustomization, it shows you the diff, you review it, then you run flux resume ks then flux suspend ks.
It's emergency handling.
And Flux is obviously prepared for emergencies, just not the way it's being requested here. GitOps is a paradigm shift, it does support "manual interventions" but it does not support "manual approvals". See the right way to handle emergencies in Stefan's comment above
The drift feature in helm-controller is very specific to how Helm works, Helm stores a fully copy of all manifests in-cluster in Kubernetes Secrets (this is called Helm storage), the drift detection runs for the storage vs cluster, not for Git vs cluster.
In kustomize-controller we don't duplicate the manifests in etcd like Helm, this would be extremely wasteful, there are no plans to do what Helm does.
the kustomize controller doesnt need to "duplicate" the manifests in etcd... it might utilize/extend the existing inventory info... even the documentation mentions drift detection: https://fluxcd.io/flux/components/kustomize/kustomizations/#inventory
but it seems this information is not exposed on the resource...
I understand it won't be a priority for you, but would you be open for PRs (that would do pretty much what the cli does if a Kustomization has a special flag).
It's emergency handling.
And Flux is obviously prepared for emergencies, just not the way it's being requested here. GitOps is a paradigm shift, it does support "manual interventions" but it does not support "manual approvals". See the right way to handle emergencies in Stefan's comment above
Those steps are manual. I'm looking for something that's easy to automate for the detection part.
We didn't designed the Flux controllers to works like a CLI in an imperative way. If you want to do that, then suspend the Flux Kustomizations and use the Flux CLI. There is a
flux diff kustomization, it shows you the diff, you review it, then you runflux resume ksthenflux suspend ks.
I have checked that, and flux diff kustomization apparently doesnt work for example if you have a distributed setup (running the flux controller in a "management" cluster and deploy the workloads into other clusters)
the kustomize controller doesnt need to "duplicate" the manifests in etcd... it might utilize/extend the existing inventory info...
Please review this message one more time, it doesn't add up. The inventory is in etcd, it's part of .status. And you can only detect a diff when you compare whole manifests, so, yes, you are basically telling us to store the manifests in etcd.
[1] Then based on the delta we could decide to update git content or let flux overwrite the drift. [2] Those steps are manual. I'm looking for something that's easy to automate for the detection part.
These two sentences also don't add up. How would you automate your decision? And CLI commands allow for a lot of bash automation, there are already commands that you can use.
I have checked that, and flux diff kustomization apparently doesnt work for example if you have a distributed setup (running the flux controller in a "management" cluster and deploy the workloads into other clusters)
This is a feature request we would be willing to accept contributions on. Let's improve the existing model, which follows GitOps principles. Let's not introduce models that go against it
Please review this message one more time, it doesn't add up. The inventory is in etcd, it's part of .status. And you can only detect a diff when you compare whole manifests, so, yes, you are basically telling us to store the manifests in etcd.
So the kustomization controller doesnt read anything from the cluster, but just blindly forcing the desired state built by kustomization build in every reconciliation?
In my mental model it could work like this:
kustomization buildto create the desired config (doesn't need to be persisted anywhere)- read the actual state from the target, which I suppose the controller anyways does to determine the sync status. (this doesn't need to be persisted)
- determine if there is any drift (delta calculation what I assume is already there in the
flux kustomization diff) - a. if automatic drift fix is enabled then do what is currently done (sync the desired state, perhaps with filtering to apply only drifted resources)
- b. if automatic drift fix is not configured (only drift detection) then record the drift in the status.inventory
These two sentences also don't add up. How would you automate your decision? And CLI commands allow for a lot of bash automation, there are already commands that you can use.
That's why I emphasized that I want to automate the detection part. With hundreds of Kustomizations it'll help a lot to know which one to look at first.
So the kustomization controller doesnt read anything from the cluster, but just blindly forcing the desired state built by kustomization build in every reconciliation?
The kustomize-controller does read from etcd. It reads the live resources. Right before applying, it diffs the live resources against the YAML built from the source, so it can apply only the resources that changed against the source. Another completely different thing is storing YAML in the cluster which is not the live resources. That's what helm does, see Stefan's comment once again:
The drift feature in helm-controller is very specific to how Helm works, Helm stores a fully copy of all manifests in-cluster in Kubernetes Secrets (this is called Helm storage), the drift detection runs for the storage vs cluster, not for Git vs cluster.
In my mental model it could work like this ...
It's not a matter of how to implement this, it's a matter of not introducing a behavior that goes against the fundamental principles of CNCF Flux
The kustomize-controller does read from etcd. It reads the live resources. Right before applying, it diffs the live resources against the YAML built from the source, so it can apply only the resources that changed against the source.
ok so basically all of the things are pretty much working the way I expected... so the only missing point is the conditional switch to actually apply the resource or just record that is drifted...
dont you think it could be at least a useful observability addition for suspended Kustomizations?
We have users that apply with a single Kustomization thousands of manifests, so we can't do what Helm does, and store all of this inside a Secret, the etcd limit of an object is 1MB. We also can't store the diff in a Kubernetes Event due to the same reason, the diff could be 1MB+. The right way to get the diff for a review is by using the Flux CLI that can stream tens of MB of YAMLs back and forth.
oh I see, I dont want the full delta, my main usecase is just to detect if the desired state is inline with the actual state or not without forcing a reconciliation. I dont want to have detailed diff stored in any kubernetes resource, only just a binary flag on the inventory entry if it is in sync or not... then if detailed diff needed kustomization diff might be triggered...
(actually this would be useful for active reconciliations as well, to indicate which resource is still out of sync)