kube
kube copied to clipboard
kube-rs
Controller continue to reconcile when object is deleted
Current and expected behavior
I use Controller to make an k8s operator. Inside my reconcile function it creates (via just PATCH) a number of resources (StatefulSets, Services, Deployments). I make all my resources to have OwnerReference to the CRD. Reconcile happens every 60s.
Sometimes when I delete CRD right before reset/refresh of the watcher happens I have my controller to stuck in reconcile loop for about 2.5 min. It receives old CRD object (from cache?) despite the fact it was deleted from k8s resources. During this time there are concurrent 2 processes happen:
- k8s tries to cleanup objects because of deleted CRD+owner reference
- my operator patches missing resources because it things CRD is in place.
Expected behavior: Controller should notice CRD object deleted and not call reconcile function.
As a side note I have questions:
- Is it okay to rely only on PATCH operations inside reconcile function?
- I don't see any option for Controller to receive
Deleted(CRD)event in the separate callback/inside reconcile function. Can I rely onObjectNotFounderror?
Possible solution
No response
Additional context
No response
Environment
Client Version: v1.22.2 Server Version: v1.20.9
Configuration and features
k8s-openapi = { version = "0.12.0", features = ["v1_20"], default-features = false }
kube = { version = "0.58.1", features = ["derive"] }
kube-derive = "0.61.0"
kube-runtime = "0.58.1"
Affected crates
No response
Would you like to work on fixing this bug?
No response
To be clear, this is about the object being controlled, not the CustomResourceDefinition itself, right? Anyway...
Sometimes when I delete CRD right before reset/refresh of the watcher happens I have my controller to stuck in reconcile loop for about 2.5 min. It receives old CRD object (from cache?) despite the fact it was deleted from k8s resources. During this time there are concurrent 2 processes happen:
k8s tries to cleanup objects because of deleted CRD+owner reference my operator patches missing resources because it things CRD is in place.
That should be fine, K8s should GC those created objects soon enough. Ultimately the race here is unavoidable, there will always be a period where clients are working with an object that has already been deleted. We might be able to reduce it, but there's no way of avoiding it completely. We could abort reconcilers that are already running, but that's already 1) best-effort only, and 2) might cause issues if we cancel in the middle of a non-atomic action. Then again, all reconcilers are subject to the ultimate cancellation anyway, so we might end up saying that this is an OK tradeoff and the reconciler should just deal with it.
Sometimes when I delete CRD right before reset/refresh of the watcher happens I have my controller to stuck in reconcile loop for about 2.5 min. It receives old CRD object (from cache?) despite the fact it was deleted from k8s resources.
Is this one reconciliation that gets stuck, or does it stop and create new reconciliations for the same object? If the latter then that's definitely a bug in kube-runtime.
Is it okay to rely only on PATCH operations inside reconcile function?
I assume you're talking about server-side apply patches here. That should be fine (and I'd generally consider that best-practice as long as you're targetting a K8s version new enough to support it). Other patch types generally only allow you to update existing objects, not create new ones.
I don't see any option for Controller to receive Deleted(CRD) event in the separate callback/inside reconcile function.
Use a finalizer (such as https://docs.rs/kube-runtime/0.63.2/kube_runtime/finalizer/fn.finalizer.html) if you want reliable deletion events.
Can I rely on ObjectNotFound error?
No.
To be clear, this is about the object being controlled, not the CustomResourceDefinition itself, right? Anyway...
yes, the part which generates CRD definition works OK.
Then again, all reconcilers are subject to the ultimate cancellation anyway, so we might end up saying that this is an OK tradeoff and the reconciler should just deal with it.
Do you mean inside the reconcile function I should query k8s API and check that there is still an object with given name/namespace? If yes, does it mean that Watcher produces unreliable results and we cannot rely only on it?
Is this one reconciliation that gets stuck, or does it stop and create new reconciliations for the same object? If the latter then that's definitely a bug in kube-runtime.
Controller produces multiple reconciliations for the object that has already been deleted. I'd expect Controller will finish the running reconciliation if any, and then will not produce any new reconciliations for the deleted object.
I assume you're talking about server-side apply patches here.
// the pattern I follow so far
fn reconcile() {
service_api.patch(...).await?;
deployment_api.patch(...).await?;
}
// vs something like this
fn reconcile() {
if service_api.get("blah-svc").await.is_some() {
service_api.update(...).await?;
}
if deployment_api.get("blah-dp").await.is_some() {
deployment_api.update(...).await?;
}
}
Do you mean inside the reconcile function I should query k8s API and check that there is still an object with given name/namespace?
No. Generally you should not need to care about this in your reconciler.
If yes, does it mean that Watcher produces unreliable results and we cannot rely only on it?
Are you getting ObjectNotFound errors for the deleted objects? If yes then watcher is fine and something funky is going on with Controller. If no then there might be a bug in there.
Either way, the intention is that watcher is reliable (but not always completely up-to-date).
Controller produces multiple reconciliations for the object that has already been deleted.
Does this happen reliably for you no matter what you do, or are you able to post a minimal example where this happens?
I'd expect Controller will finish the running reconciliation if any, and then will not produce any new reconciliations for the deleted object.
Yeah, that's the intention. If that's not the case then there is a bug somewhere that we need to find.
[snipped code snippet about patching]
The patch mode is decided by which arm of the Patch enum you pass to Api::patch (Patch::Apply, Patch::Merge, Patch::Strategic, or Patch::Json).
Does this happen reliably for you no matter what you do, or are you able to post a minimal example where this happens?
I will try to repro this. It looks it happens if I delete CRD object right before (or during?) watcher restarts list+watch operation. I'm referring to this event https://docs.rs/kube-runtime/0.64.0/kube_runtime/watcher/enum.Event.html#variant.Restarted
The patch mode is decided by which arm of the Patch enum you pass to Api::patch (Patch::Apply, Patch::Merge, Patch::Strategic, or Patch::Json).
Ah, that's right! Patch::Apply is what I use and it is Server-side apply
Are you getting ObjectNotFound errors for the deleted objects? If yes then watcher is fine and something funky is going on with Controller. If no then there might be a bug in there.
Yes, usually I receive ObjectNotFound error when I deleted CRD object.
Going to close this as it's more of a discussion issue.
For more context about ObjectNotFound on deleted owned resources see https://github.com/kube-rs/kube/issues/1167#issuecomment-1636773541 for a bigger explanation.
