Kubelet Support for Image Filesystem being split.

[kannon92]

Enhancement Description

• One-line enhancement description (can be used as a release note): Kubelet can support ImageFilesystem being separated between writeable and read-only layer

• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/4191-split-image-filesystem/README.md

• Discussion Link: Garbage Collection Working group.

• Primary contact (assignee): @kannon92

• Responsible SIGs: sig-node

• Enhancement target (which target equals to which milestone):

  • Alpha CRI and Kubelet Implementation release target (x.y): 1.29
  • Alpha Container Runtime Support: 1.30
  • Beta release target (x.y):
  • Stable release target (x.y):

• [x] Alpha - CRI Updates

  • [x] KEP (k/enhancements) update PR(s):
    • https://github.com/kubernetes/enhancements/pull/4198
  • [x] Code (k/k) update PR(s):
    • https://github.com/kubernetes/kubernetes/pull/120914
    • https://github.com/kubernetes/kubernetes/pull/120616

• [ ] Alpha - container runtime support and ecosystem

  • [x] Code (crio/crio):

    • https://github.com/cri-o/cri-o/pull/7401
    • https://github.com/cri-o/cri-o/pull/7269

  • [x] CAdvisor (google/cadvisor):

    • https://github.com/google/cadvisor/pull/3395

  • [x] CRITools (kubernetes-sigs/critools):

    • https://github.com/kubernetes-sigs/cri-tools/pull/1295

  • [x] Kubernetes (kubernetes/kubernetes):

    • https://github.com/kubernetes/kubernetes/pull/122438

  • [ ] Testing

  • https://github.com/kubernetes/test-infra/pull/31502

  • https://github.com/kubernetes/kubernetes/pull/123518

  • [ ] Docs (k/website) update PR(s):

    • https://github.com/kubernetes/website/pull/45295
    • blog post

Nice to haves:

• https://github.com/kubernetes/kubernetes/pull/123346

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

[kannon92]

/sig node

[SergeyKanzhelev]

/stage alpha /milestone v1.29 /label lead-opted-in

[kannon92]

/assign

[sreeram-venkitesh]

Hello @kannon92 👋, Enhancements team here.

Just checking in as we approach enhancements freeze on 01:00 UTC, Friday, 6th October, 2023.

This enhancement is targeting for stage alpha for v1.29 (correct me, if otherwise)

Here's where this enhancement currently stands:

• [ ] KEP readme using the latest template has been merged into the k/enhancements repo.
• [ ] KEP status is marked as implementable for latest-milestone: 1.29. KEPs targeting stable will need to be marked as implemented after code PRs are merged and the feature gates are removed.
• [ ] KEP readme has up-to-date graduation criteria
• [ ] KEP has a production readiness review that has been completed and merged into k/enhancements. (For more information on the PRR process, check here).

For this KEP https://github.com/kubernetes/enhancements/pull/4198 seems to take care of everything. Please make sure that the PR is merged in time. Will move the KEP to tracked for enhancement freeze once everything is merged into k/enhancements.

The status of this enhancement is marked as at risk for enhancement freeze. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

[sreeram-venkitesh]

Hi @kannon92, checking in once more as we approach the 1.29 enhancement freeze deadline on 01:00 UTC Friday, 6th October 2023. The status of this enhancement is marked as at risk. It looks like https://github.com/kubernetes/enhancements/pull/4198 will address most of the requirements. Please make sure that the changes are merged in time. Let me know if I missed anything. Thanks!

[kannon92]

Changes were merged so we hopefully should be good.

[npolshakova]

With KEP PR https://github.com/kubernetes/enhancements/pull/4198 approved, the enhancement is ready for the enhancements freeze. The status is now marked as tracked for enhancement freeze for 1.29. 🚀 Thank you!

[katcosgrove]

Hey there @kannon92! :wave:, v1.29 Docs Lead here. Does this enhancement work planned for v1.29 require any new docs or modification to existing docs? If so, please follows the steps here to open a PR against dev-1.29 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thursday, 19 October 2023. Also, take a look at Documenting for a release to get yourself familiarize with the docs requirement for the release. Thank you!

[katcosgrove]

Hi again @kannon92! The deadline to open a placeholder PR against k/website for required documentation is this Thursday, 19 October. Could you please update me on the status of docs for this enhancement? Thank you!

[kannon92]

This first alpha release won’t have any corresponding doc changes yet.

[James-Quigley]

Hi @kannon92 :wave: from the v1.29 Communications Release Team! We would like to check if you have any plans to publish blogs for this KEP regarding new features, removals, and deprecations for this release. If so, you need to open a PR placeholder in the website repository. The deadline will be on Tuesday 14th November 2023 (after the Docs deadline PR ready for review) Here's the 1.29 Calendar

[sreeram-venkitesh]

Hey again @kannon92 👋 v1.29 Enhancements team here,

Just checking in as we approach code freeze at 01:00 UTC Wednesday 1st November 2023 .

Here's where this enhancement currently stands:

• [x] All PRs to the Kubernetes repo that are related to your enhancement are linked in the above issue description (for tracking purposes).
• [ ] All PR/s are ready to be merged (they have approved and lgtm labels applied) by the code freeze deadline. This includes tests.

The status of this KEP is currently at risk for Code Freeze. From what I understand, https://github.com/kubernetes/kubernetes/pull/120914 and https://github.com/kubernetes/kubernetes/pull/120616 are the code PRs that is planned for the v1.29 release. Please make sure that https://github.com/kubernetes/kubernetes/pull/120616 is merged in time for the code freeze.

As always, we are here to help if any questions come up. Thanks!

[npolshakova]

With https://github.com/kubernetes/kubernetes/pull/120616 this is now marked as tracked for code freeze for 1.29! 🚀

[salehsedghpour]

/remove-label lead-opted-in

[kannon92]

Update for 1.30:

We are making good progress on the ecosystem (critools, crio and cadvisor PRs were merged). We need a cadvisor release to close out this implementation for alpha.

Will update https://github.com/kubernetes/kubernetes/pull/122438 to include cadvisor release when its available.

I have been working on e2e tests for image filesystem and I have started writing e2e configs for split disk.

[salehsedghpour]

Hello 👋 1.30 Enhancements Lead here,

I'm closing milestone 1.29 now, If you wish to progress this enhancement in v1.30, please follow the instructions here to opt in the enhancement and make sure the lead-opted-in label is set so it can get added to the tracking board and finally add /milestone v1.30. Thanks!

/milestone clear

[kannon92]

@mrunalp @SergeyKanzhelev could you add a milestone and opt in label for this feature?

I’ll be working on advisor bump and e2e tests in 1.30

[mrunalp]

/milestone v1.30 /stage alpha /label lead-opted-in

[tjons]

Hello @kannon92 👋, Enhancements team here.

Just checking in as we approach enhancements freeze on Friday, February 9th, 2024 at 02:00 UTC.

This enhancement is targeting for stage alpha for 1.30 (correct me, if otherwise)

Here's where this enhancement currently stands:

• [x] KEP readme using the latest template has been merged into the k/enhancements repo.
• [x] KEP status is marked as implementable for latest-milestone: 1.30. KEPs targeting stable will need to be marked as implemented after code PRs are merged and the feature gates are removed.
• [x] KEP readme has up-to-date graduation criteria
• [ ] KEP has a production readiness review that has been completed and merged into k/enhancements. (For more information on the PRR process, check here).

For this KEP, we would just need to complete the following:

• [ ] KEP has a production readiness review that has been completed and merged into k/enhancements. (For more information on the PRR process, check here).

The status of this enhancement is marked as at risk for enhancement freeze. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

[kannon92]

We don't need a PRR for this one. It is still staying in alpha and we had a brief one in the last release.

[kannon92]

Main goal for this next stage is to get some dependencies changes and add e2e tests. So nothing changed from first PRR review.

[tjons]

Ah, ok! Sounds good - I'll mark this as tracked for enhancements freeze! Thanks for your quick response.

[tjons]

With all the requirements fulfilled this enhancement is now marked as tracked for the upcoming enhancements freeze 🚀

[drewhagen]

Hello @kannon92 👋, 1.30 Docs Lead here.

Does this enhancement work planned for 1.30 require any new docs or modification to existing docs? If so, please follows the steps here to open a PR against dev-1.30 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thursday February 22nd 2024 18:00 PDT.

Also, take a look at Documenting for a release to get yourself familiarize with the docs requirement for the release. Thank you!

[fkautz]

Hi @kannon92

👋 from the v1.30 Communications Team! We'd love for you to opt in to write a feature blog about your enhancement!

We encourage blogs for features including, but not limited to: breaking changes, features and changes important to our users, and features that have been in progress for a long time and are graduating.

To opt in, you need to open a Feature Blog placeholder PR against the website repository. The placeholder PR deadline is 27th February, 2024. Here's the 1.30 Release Calendar

[tjons]

Hey again @kannon92 👋 Enhancements team here,

Just checking in as we approach code freeze at 02:00 UTC Wednesday 6th March 2024 .

Here's where this enhancement currently stands:

• [x] All PRs to the Kubernetes repo that are related to your enhancement are linked in the above issue description (for tracking purposes).
• [ ] All PR/s are ready to be merged (they have approved and lgtm labels applied) by the code freeze deadline. This includes tests.

For this enhancement, it looks like the following PRs are open and need to be merged before code freeze:

• https://github.com/kubernetes/kubernetes/pull/122438
• https://github.com/kubernetes/test-infra/pull/31502

Also, please let me know if there are other PRs in k/k we should be tracking for this KEP. As always, we are here to help if any questions come up. Thanks!

[tjons]

Hey @kannon92 - looks like the two above PRs merged! I'm seeing that https://github.com/kubernetes/kubernetes/pull/123518 has been added to the issue... code freeze is ~6 hours away. Do you think it will merge in time?

[kannon92]

This is for test freeze. So I don’t think it should be tracked in code freeze.

[salehsedghpour]

Hello @kannon92 👋 , Enhancements team here.

With all the implementation(code related) PRs merged as per the issue description:

• https://github.com/kubernetes/kubernetes/pull/122438
• https://github.com/kubernetes/test-infra/pull/31502

This enhancement is now marked as tracked for code freeze for the v1.30 Code Freeze!

[kannon92]

We hit a blocker with adding tests. It turns out that there is a fix in container/storage which crio uses for this work.

We are waiting for podman 5.0 to be released and then we can update crio to use the latest storage changes. We want to kick off a branch for 1.30 that does not include podman 5.0 changes so we are going to wait until 1.31 to get the e2e tests working.

So I will not be documenting this feature in a blog post because there is still some risk with the feature and I do not want to advertise the feature yet.

Kubelet changes are good as is. This is mainly a problem in the container runtime so we do not need any revert or anything.