Feature Request: Suppress `the object has been modified` output by default

[kahirokunn]

/area monitoring

Describe the feature

Overview

In Knative Serving, the "the object has been modified" logs and events that occur during object updates often only result in retries within the processing logic, creating significant noise due to frequent output. Since these are typically unnecessary during normal operations, could we modify them to not appear at standard log levels and only be displayed when needed in Debug mode (or Verbose log level)?

Why is this needed

• In Knative Serving, object update conflicts are often resolved through retries without additional processing or errors occurring.
• Unnecessary logs and events may obscure information that actually needs attention.
• Currently, operators spend extra time reviewing unnecessary logs, which could also impact monitoring/alert settings.

Proposal

• Suppress or hide by default (at a lower log level) the "the object has been modified" logs and events, only displaying them when log levels are explicitly increased.
• Output detailed information only when necessary (for example, when retries fail after a certain number of attempts).

Expected Benefits

• Reduced noisy log output, leading to lower operational overhead.
• Easier identification of truly necessary logs and events without missing important information.
• Ability to explicitly enable Debug/Verbose mode only when detailed logs are needed during error scenarios.

We would greatly appreciate your consideration of adding this functionality. Thank you for your consideration.

[dprotaso]

which controller/reconciler do you see this happening with the most?

[kahirokunn]

I think there were a lot of outputs from multiple controllers, but I will check again.

[github-actions[bot]]

This issue is stale because it has been open for 90 days with no activity. It will automatically close after 30 more days of inactivity. Reopen the issue with /reopen. Mark the issue as fresh by adding the comment /remove-lifecycle stale.

[dprotaso]

Are you using net-istio by any chance?

[kahirokunn]

No, I use contour.

[github-actions[bot]]

This issue is stale because it has been open for 90 days with no activity. It will automatically close after 30 more days of inactivity. Reopen the issue with /reopen. Mark the issue as fresh by adding the comment /remove-lifecycle stale.

[kahirokunn]

/reopen

[knative-prow[bot]]

@kahirokunn: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[linkvt]

I collected the logs of a simple KService creation that contained the "the object has been modified" output:

controller-9b54b47b7-blv4f controller {"severity":"ERROR","timestamp":"2025-11-12T14:26:05.209390667Z","logger":"controller","caller":"controller/controller.go:545","message":"Reconcile error","commit":"78814e1-dirty","knative.dev/pod":"controller-9b54b47b7-blv4f","knative.dev/controller":"knative.dev.serving.pkg.reconciler.labeler.Reconciler","knative.dev/kind":"serving.knative.dev.Route","knative.dev/traceid":"4147c4cf-dd0c-40b8-89ff-8d87d5785ded","knative.dev/key":"default/whoami","duration":"16.664444ms","error":"failed to set finalizers: Operation cannot be fulfilled on routes.serving.knative.dev \"whoami\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/[email protected]/controller/controller.go:545\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/[email protected]/controller/controller.go:522\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/[email protected]/controller/controller.go:478"}
controller-9b54b47b7-blv4f controller {"severity":"ERROR","timestamp":"2025-11-12T14:26:05.21964462Z","logger":"controller","caller":"controller/controller.go:545","message":"Reconcile error","commit":"78814e1-dirty","knative.dev/pod":"controller-9b54b47b7-blv4f","knative.dev/controller":"knative.dev.serving.pkg.reconciler.labeler.Reconciler","knative.dev/kind":"serving.knative.dev.Route","knative.dev/traceid":"d47a0f1e-4556-4ada-b4cf-766c6a4ccaa0","knative.dev/key":"default/whoami","duration":"3.537333ms","error":"failed to set finalizers: Operation cannot be fulfilled on routes.serving.knative.dev \"whoami\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/[email protected]/controller/controller.go:545\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/[email protected]/controller/controller.go:522\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/[email protected]/controller/controller.go:478"}
controller-9b54b47b7-blv4f controller {"severity":"ERROR","timestamp":"2025-11-12T14:26:05.782175472Z","logger":"controller","caller":"revision/reconciler.go:303","message":"Returned an error","commit":"78814e1-dirty","knative.dev/pod":"controller-9b54b47b7-blv4f","knative.dev/controller":"knative.dev.serving.pkg.reconciler.revision.Reconciler","knative.dev/kind":"serving.knative.dev.Revision","knative.dev/traceid":"7279be8e-b170-4a12-be49-f43945b5e838","knative.dev/key":"default/whoami-00001","targetMethod":"ReconcileKind","error":"failed to update deployment \"whoami-00001-deployment\": Operation cannot be fulfilled on deployments.apps \"whoami-00001-deployment\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"knative.dev/serving/pkg/client/injection/reconciler/serving/v1/revision.(*reconcilerImpl).Reconcile\n\tknative.dev/serving/pkg/client/injection/reconciler/serving/v1/revision/reconciler.go:303\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/[email protected]/controller/controller.go:521\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/[email protected]/controller/controller.go:478"}
controller-9b54b47b7-blv4f controller {"severity":"ERROR","timestamp":"2025-11-12T14:26:05.782251181Z","logger":"controller","caller":"controller/controller.go:545","message":"Reconcile error","commit":"78814e1-dirty","knative.dev/pod":"controller-9b54b47b7-blv4f","knative.dev/controller":"knative.dev.serving.pkg.reconciler.revision.Reconciler","knative.dev/kind":"serving.knative.dev.Revision","knative.dev/traceid":"7279be8e-b170-4a12-be49-f43945b5e838","knative.dev/key":"default/whoami-00001","duration":"15.273345ms","error":"failed to update deployment \"whoami-00001-deployment\": Operation cannot be fulfilled on deployments.apps \"whoami-00001-deployment\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/[email protected]/controller/controller.go:545\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/[email protected]/controller/controller.go:522\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/[email protected]/controller/controller.go:478"}
autoscaler-7d65fbcf7d-snpdd autoscaler {"severity":"ERROR","timestamp":"2025-11-12T14:26:09.619578349Z","logger":"autoscaler","caller":"podautoscaler/reconciler.go:314","message":"Returned an error","commit":"2f3129a","knative.dev/controller":"knative.dev.serving.pkg.reconciler.autoscaling.kpa.Reconciler","knative.dev/kind":"autoscaling.internal.knative.dev.PodAutoscaler","knative.dev/traceid":"e2133325-d47a-4b3c-bc3f-3fd57e8d73a7","knative.dev/key":"default/whoami-00001","targetMethod":"ReconcileKind","error":"error reconciling SKS: error updating SKS whoami-00001: Operation cannot be fulfilled on serverlessservices.networking.internal.knative.dev \"whoami-00001\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"knative.dev/serving/pkg/client/injection/reconciler/autoscaling/v1alpha1/podautoscaler.(*reconcilerImpl).Reconcile\n\tknative.dev/serving/pkg/client/injection/reconciler;/autoscaling/v1alpha1/podautoscaler/reconciler.go:314\nmain.(*leaderAware).Reconcile\n\tknative.dev/serving/cmd/autoscaler/leaderelection.go:44\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/[email protected]/controller/controller.go:521\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/[email protected]/controller/controller.go:478"}
autoscaler-7d65fbcf7d-snpdd autoscaler {"severity":"ERROR","timestamp":"2025-11-12T14:26:09.619650808Z","logger":"autoscaler","caller":"controller/controller.go:545","message":"Reconcile error","commit":"2f3129a","knative.dev/controller":"knative.dev.serving.pkg.reconciler.autoscaling.kpa.Reconciler","knative.dev/kind":"autoscaling.internal.knative.dev.PodAutoscaler","knative.dev/traceid":"e2133325-d47a-4b3c-bc3f-3fd57e8d73a7","knative.dev/key":"default/whoami-00001","duration":"12.223309ms","error":"error reconciling SKS: error updating SKS whoami-00001: Operation cannot be fulfilled on serverlessservices.networking.internal.knative.dev \"whoami-00001\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/[email protected]/controller/controller.go:545\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/[email protected]/controller/controller.go:522\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/[email protected]/controller/controller.go:478"}
controller-9b54b47b7-blv4f controller {"severity":"ERROR","timestamp":"2025-11-12T14:26:09.622113462Z","logger":"controller","caller":"revision/reconciler.go:303","message":"Returned an error","commit":"78814e1-dirty","knative.dev/pod":"controller-9b54b47b7-blv4f","knative.dev/controller":"knative.dev.serving.pkg.reconciler.revision.Reconciler","knative.dev/kind":"serving.knative.dev.Revision","knative.dev/traceid":"d67bcaa0-b06a-473c-bb90-b069797ebfa0","knative.dev/key":"default/whoami-00001","targetMethod":"ReconcileKind","error":"failed to update deployment \"whoami-00001-deployment\": Operation cannot be fulfilled on deployments.apps \"whoami-00001-deployment\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"knative.dev/serving/pkg/client/injection/reconciler/serving/v1/revision.(*reconcilerImpl).Reconcile\n\tknative.dev/serving/pkg/client/injection/reconciler/serving/v1/revision/reconciler.go:303\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/[email protected]/controller/controller.go:521\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/[email protected]/controller/controller.go:478"}
controller-9b54b47b7-blv4f controller {"severity":"ERROR","timestamp":"2025-11-12T14:26:09.622308423Z","logger":"controller","caller":"controller/controller.go:545","message":"Reconcile error","commit":"78814e1-dirty","knative.dev/pod":"controller-9b54b47b7-blv4f","knative.dev/controller":"knative.dev.serving.pkg.reconciler.revision.Reconciler","knative.dev/kind":"serving.knative.dev.Revision","knative.dev/traceid":"d67bcaa0-b06a-473c-bb90-b069797ebfa0","knative.dev/key":"default/whoami-00001","duration":"16.745029ms","error":"failed to update deployment \"whoami-00001-deployment\": Operation cannot be fulfilled on deployments.apps \"whoami-00001-deployment\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/[email protected]/controller/controller.go:545\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/[email protected]/controller/controller.go:522\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/[email protected]/controller/controller.go:478"}

There was a PR quite a few years ago that implemented retries only for status updates, I guess this means that we do not want retries for non-status updates.

We could thus update the controller error handling to simply requeue in case of conflicts with a debug log message instead of throwing the error here: https://github.com/knative/pkg/blob/5037c47608290c30e56fa66e0fcc1de8b040c865/controller/controller.go#L545

I'll look into it and check if my idea is feasible, feedback is still appreciated.

/assign

[dprotaso]

@linkvt merged a PR in pkg so the logs are less verbose

🎉

[dprotaso]

Going to close this out

[kahirokunn]

Thank you very much 🥺