Kubelet limit of Parallel Image Pulls

[ruiwen-zhao]

Enhancement Description

• One-line enhancement description (can be used as a release note): Add a node-level limit to kubelet to limit the number of parallel image pulls

• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/3673-kubelet-parallel-image-pull-limit/README.md

• Discussion Link:

• Primary contact (assignee): @pacoxu @ruiwen-zhao

• Responsible SIGs: SIG Node

• Enhancement target (which target equals to which milestone):

  • Alpha release target (x.y): 1.27
  • Beta release target (x.y): 1.30
  • Stable release target (x.y):

• [x] Alpha

  • [x] KEP (k/enhancements) update PR(s):
    • https://github.com/kubernetes/enhancements/pull/3713
  • [x] Code (k/k) update PR(s):
    • https://github.com/kubernetes/kubernetes/pull/115220
  • [x] Docs (k/website) update PR(s):
    • https://github.com/kubernetes/website/pull/39732

• [ ] Beta

  • [ ] KEP (k/enhancements) update PR(s):
    • [x] 1.28 https://github.com/kubernetes/enhancements/pull/4036;
    • [x] 1.29 https://github.com/kubernetes/enhancements/pull/4173
    • [x] 1.30 https://github.com/kubernetes/enhancements/pull/4452
  • [ ] Code (k/k) update PR(s): https://github.com/kubernetes/kubernetes/pull/121604
  • [ ] Docs (k/website) update(s):
    • [x] ~~https://github.com/kubernetes/website/pull/42061(mistakenly merged in v1.28, and reverted).~~

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

[ruiwen-zhao]

/sig node

[pacoxu]

Some intial ideas can be found in https://docs.google.com/document/d/1MMZlKSxkDBDz0kEn9JGm848_IJgydJqXWz5t9D3Ev8I/edit# and https://github.com/kubernetes/kubernetes/pull/112242 https://github.com/kubernetes/kubernetes/pull/112044

[wojtek-t]

/lead opted-in /milestone v1.27

[wojtek-t]

/label lead-opted-in

[marosset]

Hello @pacoxu & @ruiwen-zhao 👋, Enhancements team here.

Just checking in as we approach enhancements freeze on 18:00 PDT Thursday 9th February 2023.

This enhancement is targeting for stage alpha for v1.27 (correct me, if otherwise)

Here's where this enhancement currently stands:

• [ ] KEP readme using the latest template has been merged into the k/enhancements repo.
• [ ] KEP status is marked as implementable for latest-milestone: v1.27
• [ ] KEP readme has a updated detailed test plan section filled out
• [ ] KEP readme has up to date graduation criteria
• [ ] KEP has a production readiness review that has been completed and merged into k/enhancements.

For this enhancement, it looks like https://github.com/kubernetes/enhancements/pull/3713 will address most of these requirements. Please be sure to also:

• Update the status to implemnetable in the kep.yaml (when ready)

The status of this enhancement is marked as at risk. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

[ruiwen-zhao]

Hello @pacoxu & @ruiwen-zhao 👋, Enhancements team here.

Just checking in as we approach enhancements freeze on 18:00 PDT Thursday 9th February 2023.

This enhancement is targeting for stage alpha for v1.27 (correct me, if otherwise)

Here's where this enhancement currently stands:

• [ ] KEP readme using the latest template has been merged into the k/enhancements repo.
• [ ] KEP status is marked as implementable for latest-milestone: v1.27
• [ ] KEP readme has a updated detailed test plan section filled out
• [ ] KEP readme has up to date graduation criteria
• [ ] KEP has a production readiness review that has been completed and merged into k/enhancements.

For this enhancement, it looks like #3713 will address most of these requirements. Please be sure to also:

• Update the status to implemnetable in the kep.yaml (when ready)

The status of this enhancement is marked as at risk. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

Updated the KEP PR https://github.com/kubernetes/enhancements/pull/3713 to change the status to implemnetable. Once it is merged then all of the requirements will be addressed.

[marosset]

Thanks @ruiwen-zhao - this enhancement is ready to be tracked for v1.27

[pacoxu]

/assign @pacoxu @ruiwen-zhao

[ruiwen-zhao]

Draft document update: https://github.com/kubernetes/website/pull/39732

[mickeyboxell]

Hi @ruiwen-zhao @pacoxu 👋, I’m reaching out from the 1.27 Release Docs team. This enhancement is marked as ‘Needs Docs’ for the 1.27 release.

Please follow the steps detailed in the documentation to open a PR against dev-1.27 branch in the k/website repo. This PR can be just a placeholder at this time, and must be created by March 16. For more information, please take a look at Documenting for a release to familiarize yourself with the documentation requirements for the release.

Please feel free to reach out with any questions. Thanks!

[ruiwen-zhao]

Hi @ruiwen-zhao @pacoxu 👋, I’m reaching out from the 1.27 Release Docs team. This enhancement is marked as ‘Needs Docs’ for the 1.27 release.

Please follow the steps detailed in the documentation to open a PR against dev-1.27 branch in the k/website repo. This PR can be just a placeholder at this time, and must be created by March 16. For more information, please take a look at Documenting for a release to familiarize yourself with the documentation requirements for the release.

Please feel free to reach out with any questions. Thanks!

Is https://github.com/kubernetes/website/pull/39732 sufficient for now?

[marosset]

Hi @ruiwen-zhao @pacoxu :wave:,

Checking in as we approach 1.27 code freeze at 17:00 PDT on Tuesday 14th March 2023.

Please ensure the following items are completed:

• [x] All PRs to the Kubernetes repo that are related to your enhancement are linked in the above issue description (for tracking purposes).
• [x] All PRs are fully merged by the code freeze deadline.

Please let me know if there are any other PRs in k/k I should be tracking for this KEP. I see the following PR was closed (https://github.com/kubernetes/kubernetes/pull/112242) - Is it going to be replaced with another PR?

As always, we are here to help should questions come up. Thanks!

[ruiwen-zhao]

Hi @marosset

The PR is https://github.com/kubernetes/kubernetes/pull/115220, which has been merged already. Thanks!

[SergeyKanzhelev]

this will be progressing to beta in 1.28 as per the sig node meeting 5/2/2023

/milestone v1.28

[pacoxu]

this will be progressing to beta in 1.28 as per the sig node meeting 5/2/2023

I did not attend the meeting. @ruiwen-zhao could you share some details about the plan of this feature in v1.28?

[ruiwen-zhao]

this will be progressing to beta in 1.28 as per the sig node meeting 5/2/2023

I did not attend the meeting. @ruiwen-zhao could you share some details about the plan of this feature in v1.28?

I am thinking of adding a e2e test for parallel image pulling with a MaxParallelImagePulls set. And that will be the only criteria for graduating to beta. Plus GKE starts using this feature in 1.27, so we could gather some feedback if this feature causes any regression.

WDYT?

[pacoxu]

If GKE will use it， the feedback would be very valuable.

To me， the only unsure thing about enabling parallel image pulling is if disk will be overloaded.

[ruiwen-zhao]

If GKE will use it， the feedback would be very valuable.

To me， the only unsure thing about enabling parallel image pulling is if disk will be overloaded.

Yep, we (GKE) have done some experiment with different parallelism and observed disk throttling.

One more thing, I don't plan to enable parallel pull by default as part of beta promotion. The beta promotion of this KEP is mostly to tell people this feature is more stable now.

[ruiwen-zhao]

/milestone v1.28

[k8s-ci-robot]

@ruiwen-zhao: You must be a member of the kubernetes/milestone-maintainers GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your Milestone Maintainers Team and have them propose you as an additional delegate for this responsibility.

In response to this:

/milestone v1.28

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[SergeyKanzhelev]

it is already in the milestone

[wojtek-t]

it is already in the milestone

@SergeyKanzhelev It's not in the tracking board https://github.com/orgs/kubernetes/projects/140/views/2 And i'ts because it's marked as Alpha at this point, and is not lead-opted-in for the release. Setting milestone is not enough - you need to lead-opt-in it.

[SergeyKanzhelev]

/label lead-opt-in

[k8s-ci-robot]

@SergeyKanzhelev: The label(s) /label lead-opt-in cannot be applied. These labels are supported: api-review, tide/merge-method-merge, tide/merge-method-rebase, tide/merge-method-squash, team/katacoda, refactor, lead-opted-in, tracked/no, tracked/out-of-tree, tracked/yes. Is this label configured under labels -> additional_labels or labels -> restricted_labels in plugin.yaml?

In response to this:

/label lead-opt-in

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[SergeyKanzhelev]

/label lead-opted-in

[SergeyKanzhelev]

/stage beta

[ruheenaansari34]

Hello @ruiwen-zhao 👋, 1.28 Enhancements team here.

Just checking in as we approach enhancements freeze on 1:00 UTC on Friday 16th June 2023.

This enhancement is targeting stage beta for 1.28 (correct me, if otherwise)

Here's where this enhancement currently stands:

• [x] KEP readme using the latest template has been merged into the k/enhancements repo.
• [x] KEP status is marked as implementable for latest-milestone: 1.28
• [ ] KEP readme has a updated detailed test plan section filled out
• [x] KEP readme has up-to-date graduation criteria
• [x] KEP has a production readiness review that has been completed and merged into k/enhancements.

For this KEP, we would need to take care of:

• Updating the Test plan section, especially accepting the acknowledgment specified here

The status of this enhancement is marked as at risk. Please keep the issue description up-to-date with the appropriate stages as well. Thank you!

[ruiwen-zhao]

Hi @ruheenaansari34 , i have sent https://github.com/kubernetes/enhancements/pull/4086 to accept the acknowledgement.

[Atharva-Shinde]

Hey @ruiwen-zhao

With all the KEP requirements in place and merged into k/enhancements, this enhancement is all good for the upcoming enhancements freeze. 🚀 The status of this enhancement is marked as tracked. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

[AdminTurnedDevOps]

Hey @ruiwen-zhao

1.28 Docs Shadow here.

Does this enhancement work planned for 1.28 require any new docs or modification to existing docs?

If so, please follows the steps here to open a PR against dev-1.28 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thursday 20th July 2023.

Also, take a look at Documenting for a release to get yourself familiarize with the docs requirement for the release.

Thank you!