Stale Controller Mitigation

[michaelasp]

Enhancement Description

• One-line enhancement description (can be used as a release note): Add metrics to track stale KCM controllers
• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/blob/master/keps/sig-api-machinery/5647-stale-controller-handling/README.md
• Discussion Link: Sig API Machinery https://docs.google.com/document/d/1x9RNaaysyO0gXHIr1y50QFbiL1x8OWnk2v3XnrdkT5Y/edit?tab=t.0#heading=h.hw1pc91j5zaj
• PRs by stage and milestone:
  • [ ] Alpha - v1.35
    • [x] KEP (k/enhancements) update PR(s):
      • [x] #5649
    • [ ] Code (k/k) update PR(s):
      • [ ] https://github.com/kubernetes/kubernetes/pull/134827
      • [ ] https://github.com/kubernetes/kubernetes/pull/134762
      • [ ] https://github.com/kubernetes/kubernetes/pull/134937
    • [ ] Docs (k/website) update PR(s):
      • [ ] https://github.com/kubernetes/website/pull/52901

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

[michaelasp]

/sig api-machinery

[michaelasp]

/assign @michaelasp

[jpbetz]

/label lead-opted-in

[serathius]

KEP was merged as implementable state with alpha targeting 1.35

/stage alpha /milestone v1.35

[rayandas]

Hey folks, we're evaluating the exception request at the moment and we will get back to you. Thanks!

[rayandas]

The exception request has been approved. This KEP is now Tracked for Enhancements Freeze. Thanks!

[rayandas]

/label tracked/yes

[aakankshabhende]

Hello @michaelasp 👋, this is Aakanksha (@aakankshabhende) from the v1.35 Communications Team!

For the v1.35 release, we are currently in the process of collecting and curating a list of potential feature blogs, and we'd love for you to consider writing one for your enhancement!

As you may be aware, feature blogs are a great way to communicate to users about features which fall into (but not limited to) the following categories:

• This introduces some breaking change(s)
• This has significant impacts and/or implications to users
• ...Or this is a long-awaited feature, which would go a long way to cover the journey more in detail 🎉

To opt in to write a feature blog, could you please let us know and open a "Feature Blog placeholder PR" (which can be only a skeleton at first) against the website repository by Friday, 31st October? For more information about writing a blog, please find the blog contribution guidelines 📚

[!Tip] Some timeline to keep in mind:

• 12:00 UTC Friday, 31st October: Feature blog PR freeze
• Friday, 21st November: Feature blogs ready for review
• You can find more in the release document

[!Note] In your placeholder PR, use XX characters for the blog date in the front matter and file name. We will work with you on updating the PR with the publication date once we have a final number of feature blogs for this release.

[Urvashi0109]

Hello @michaelasp👋, v1.35 Docs Lead here.

Does this enhancement work planned for v1.35 require any new docs or modification to existing docs? If so, please follow the steps here to open a PR against dev-1.35 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thursday 23th October 2025. Also, take a look at Documenting for a release to get yourself familiarize with the docs requirement for the release.

Thank you!

[Urvashi0109]

Hello @michaelasp👋, v1.35 Docs Lead here.

Just Reminder! If enhancement work planned for v1.35 require any new docs or modification to existing docs, please follow the steps here to open a PR against dev-1.35 branch in the k/website repo.

This PR can be just a placeholder at this time and must be created before Thursday 23th October 2025.

Also, take a look at Documenting for a release to get yourself familiarize with the docs requirement for the release.

Thank you!

[Urvashi0109]

Hey @michaelasp👋, we need PR against dev-1.35 branch in the k/website repo. Please follow the steps here to open a PR

[michaelasp]

Hey @Urvashi0109! I don't think we need any new documentation for an alpha feature, lmk if we need to document anything though. Thanks!

[dipesh-rawat]

Hi @michaelasp 👋, v1.35 Release Team shadow here!

From reviewing the KEP-5647 readme (here), I noticed that it aims introduces the feature behind an alpha feature gate StaleControllerConsistency. We'll need to ensure this is documented in the feature gate documentation (here). I also saw another feature gate MonitorInformerStaleness introduced in the code PR https://github.com/kubernetes/kubernetes/pull/134762 - that one will need to be documented as well.

For more details, please refer to the documentation here.

When you have a moment, could you please update the issue description (here) to include:

• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/blob/master/keps/sig-api-machinery/5647-stale-controller-handling/README.md)
• Related code PRs and docs PRs (whenever PRs are created)

This helps the release team track everything smoothly for this KEP.

[michaelasp]

Gotcha, will send out a draft PR by EOD

[sayanchowdhury]

Thanks @michaelasp for creating the draft PR but currently the PR is created against main. Could you please to update the PR against dev-1.35? You can refer to this link for help.

[michaelasp]

Thanks [@michaelasp](https://github.com/michaelasp) for creating the draft PR but currently the PR is created against main. Could you please to update the PR against dev-1.35? You can refer to this link for help.

Yep done, just accidentally opened against the wrong branch

[aakankshabhende]

Hi @michaelasp👋, this is Aakanksha (@aakankshabhende) from v1.35 Communications Team here again!

This is a gentle reminder for the feature blog deadline mentioned above, which is 12:00 UTC Friday, 31st October. To opt in, please let us know and open a Feature Blog placeholder PR against k/website by the deadline. If you have any questions, please feel free to reach out to us!

[!Tip] Some timeline to keep in mind:

• 12:00 UTC Friday, 31st October: Feature blog PR freeze
• Friday, 21st November: Feature blogs ready for review
• You can find more in the release document

[!Note] In your placeholder PR, use XX characters for the blog date in the front matter and file name. We will work with you on updating the PR with the publication date once we have a final number of feature blogs for this release.

[xigang]

Thanks to @michaelasp for the contribution — this KEP addresses an issue I previously encountered: https://github.com/kubernetes/kubernetes/issues/129795

[rayandas]

Hey again @michaelasp 👋, v1.35 Enhancements team here,

Just checking in as we approach code freeze and test freeze on Thursday 6th November 2025 (AoE) / Friday 7th November 2025, 12:00 UTC.

Here's where this enhancement currently stands:

• [ ] All PRs to the Kubernetes repo that are related to your enhancement are linked in the above issue description (for tracking purposes).
• [ ] All PRs are ready to be merged (they have approved and lgtm labels applied) by the code freeze deadline. This includes tests.

Per the issue description, these are all of the implementation (code-related) PRs for v1.35, some of which are not merged yet:

• https://github.com/kubernetes/kubernetes/pull/134937 (Please add it in the KEP description)
• https://github.com/kubernetes/kubernetes/pull/134827
• https://github.com/kubernetes/kubernetes/pull/134762

Please let me know (and keep the issue description updated) if there are any other PRs in k/k that we should track for this KEP, so that we can maintain accurate status.

If the implementation work for this enhancement is occurring out-of-tree (i.e., outside of k/k), please link the relevant PRs in the issue description for visibility. Alternatively, if you're unable to provide specific PR links, a confirmation that all out-of-tree implementation work is complete and merged will help us finalize tracking and maintain accuracy.

The status of this enhancement is marked as At risk for code freeze.

If you anticipate missing code freeze, you can file an exception request in advance.

[michaelasp]

Should hopefully be good, most of the code seems to be in a good spot, just needs final rounds of review and approval.

[michaelasp]

Probably going to have to push this to 1.36 unfortunately 😢, likely won't make it before code freeze. From discussing with @serathius and @liggitt there are still some open questions on how to best expose functionality in client-go for controller usage and without some proper thought may lead to incorrect decisions/assumptions that we expose to end users.

[rayandas]

Hello @michaelasp 👋, v1.35 Enhancements team here,

Unfortunately, the implementation (code-related) PRs associated with this enhancement are not in the merge-ready state by code-freeze and hence this enhancement is now removed from the v1.35 milestone.

Additionally, if any of the merged implementation PRs for this enhancement include user-facing changes, please let us know. This will help us determine whether the changes should be documented or considered for rollback to maintain release integrity.

If you still wish to progress this enhancement in v1.35, please file an exception request as soon as possible, within three days. If you have any questions, you can reach out in the #release-enhancements channel on Slack and we'll be happy to help. Thanks!

/label tracked/no /milestone clear

[rayandas]

Hi @michaelasp 👋 v1.35 Enhancements Team here.

We are nearing the close of the v1.35 release. As part of the end of release cleanup, we are removing the lead-opted-in label to prepare for the next cycle.

If you would like this enhancement to be considered for the next release, please have the SIG lead opt in again by re adding the label and setting the appropriate milestone for the new release. This ensures the enhancement appears on the tracking board.

Thanks!

/remove-label lead-opted-in

[jpbetz]

/milestone v1.36 /label lead-opted-in

[k8s-ci-robot]

@jpbetz: The provided milestone is not valid for this repository. Milestones in this repository: [v1.25, v1.27, v1.28, v1.29, v1.30, v1.31, v1.32, v1.33, v1.34, v1.35]

Use /milestone clear to clear the milestone.

In response to this:

/milestone v1.36 /label lead-opted-in

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[rayandas]

Hi @jpbetz the new milestone is not yet created. It will be created after the v1.35 release (which is today 🎉 ). Until then I am clearing the lead-opted-in label.

/remove-label lead-opted-in

You can add it after today's release when the new milestone is created.