controller-runtime icon indicating copy to clipboard operation
controller-runtime copied to clipboard

rateLimiter resets after resource creation, but should after reconcile end

Open setood opened this issue 2 years ago • 4 comments

i have an operator, that creates service and pod for my custom resource. if i get an error immediately, the rateLimiter works as expected:

pod.Name = "!!!"
if err := r.Create(ctx, pod); err != nil { // error
    return ctrl.Result{}, err
}

but if i create an object before that, the rateLimiter is reset and the reconsil starts to be called non-stop:

if err := r.Create(ctx, service); err != nil { //successfull
    return ctrl.Result{}, err
}

pod.Name = "!!!"
if err := r.Create(ctx, pod); err != nil { // error
    return ctrl.Result{}, err
}

how can i workaround this?

setood avatar Mar 25 '22 16:03 setood

This is likely because you have a handler that reacts to the service creation, and since it takes a moment for the event to arrive, it does so after the error occurred and resets the rate limiter.

There is nothing specific you should do, just keep your reconciler idempotent. This will cause it to not create a service the second time around, fail the pod creation and then go into back off as expected.

alvaroaleman avatar Apr 06 '22 00:04 alvaroaleman

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle stale
  • Mark this issue or PR as rotten with /lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot avatar Jul 05 '22 01:07 k8s-triage-robot

@setood For this case, you should use IsAlreadyExists to check the error returned from Create:

if err := r.Create(ctx, service); err != nil {
    if !errors.IsAlreadyExists(err) {
        return ctrl.Result{}, err
    }
    // maybe get the object if you need
}

pod.Name = "!!!"
if err := r.Create(ctx, pod); err != nil { // error
    if !errors.IsAlreadyExists(err) {
        return ctrl.Result{}, err
    }
    // maybe get the object if you need
}

So that even if it failed to create pod in the first reconcile, it will not return error of creating service in the next reconcile.

FillZpp avatar Jul 05 '22 03:07 FillZpp

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot avatar Aug 04 '22 04:08 k8s-triage-robot