Kubernetes VolumeAttributesClass ModifyVolume

[sunnylovestiramisu]

Enhancement Description

• One-line enhancement description (can be used as a release note): An extension to the Kubernetes Persistent Volume API to allow users to dynamically control volume options iops + throughput.

• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/3751-volume-attributes-class/README.md

• Discussion Link:

• CSI Spec for ModifyVolume
• keps/sig-storage/3751-volume-attributes-class
• https://github.com/kubernetes/enhancements/pull/4116

• Primary contact (assignee): @mattcary @sunnylovestiramisu

• Responsible SIGs: sig-storage

• Enhancement target (which target equals to which milestone):

  • Alpha release target (x.y): 1.29
  • Beta release target (x.y): 1.31
  • Stable release target (x.y): 1.3x

• [x] Alpha

  • [x] KEP (k/enhancements) update PR(s):
    • [x] https://github.com/kubernetes/enhancements/pull/3780
    • [x] https://github.com/kubernetes/enhancements/pull/4245
    • [x] https://github.com/kubernetes/enhancements/pull/4298
  • [x] Code (k/k) update PR(s):
    • [x] v1.29: https://github.com/kubernetes/kubernetes/pull/121104
    • [x] https://github.com/kubernetes-csi/external-provisioner/pull/1068
  • [x] Docs (k/website) update PR(s):
    • [x] v1.29: https://github.com/kubernetes/website/pull/43463
    • [x] https://github.com/kubernetes/website/pull/43777

• [ ] Beta

  • [x] KEP (k/enhancements) update PR(s): https://github.com/kubernetes/enhancements/pull/4698
  • [ ] Code (k/k) update PR(s): https://github.com/kubernetes/kubernetes/pull/123549, https://github.com/kubernetes/kubernetes/pull/121902
  • [ ] Docs (k/website) update(s): https://github.com/kubernetes/website/pull/46785

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

[sunnylovestiramisu]

/sig storage

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[sunnylovestiramisu]

/remove-lifecycle stale

[salehsedghpour]

Hello @sunnylovestiramisu 👋, Enhancements team here.

Just checking in as we approach enhancements freeze on 01:00 UTC Friday, 16th June 2023.

This enhancement is targeting for stage alpha for 1.28 (correct me, if otherwise)

Here's where this enhancement currently stands:

• [X] KEP readme using the latest template has been merged into the k/enhancements repo.
• [ ] KEP status is marked as implementable for latest-milestone: 1.28
• [ ] KEP readme has a updated detailed test plan section filled out
• [X] KEP readme has up to date graduation criteria
• [ ] KEP has a production readiness review that has been completed and merged into k/enhancements.

For this KEP, we would just need to update the following:

• Update status to implementable in kep.yaml
• Add the test plan in the readme file
• Address questions inside the Production Readiness Review Questionnaire

The status of this enhancement is marked as at risk. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

[salehsedghpour]

Hi @sunnylovestiramisu 👋, just checking in before the enhancements freeze on 01:00 UTC Friday, 16th June 2023.

The status for this enhancement is at risk.

For this KEP, we would just need to update the following:

• Update status to implementable in kep.yaml
• Add the test plan in the readme file
• Address questions inside the Production Readiness Review Questionnaire

Let me know if I missed anything. Thanks!

[AdminTurnedDevOps]

Hey @sunnylovestiramisu

1.28 Docs Shadow here.

Does this enhancement work planned for 1.28 require any new docs or modification to existing docs?

If so, please follows the steps here to open a PR against dev-1.28 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thursday 20th July 2023.

Also, take a look at Documenting for a release to get yourself familiarize with the docs requirement for the release.

Thank you!

[sunnylovestiramisu]

@AdminTurnedDevOps I think we just need a placeholder PR so that the docs team can add it to tracking. I will follow the instructions, thanks!

[carlory]

Hi @sunnylovestiramisu , Could you update the link of Kubernetes Enhancement Proposal to https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/3751-volume-attributes-class.

The pr link is difficult for users to learn the content of this KEP.

[salehsedghpour]

Hey again @sunnylovestiramisu :wave: Just checking in as we approach Code freeze at 01:00 UTC Friday, 19th July 2023 .

I don't see any code (k/k) update PR(s) in the issue description so if there are any k/k related PR(s) that we should be tracking for this KEP please link them in the issue description above.

As always, we are here to help if any questions come up. Thanks!

[sunnylovestiramisu]

@salehsedghpour So far there was only one PR in review and I discussed with sig-storage lead @msau42 from now on we want to make sure the release in k8s and csi side cars all be ready before merging any PRs.

We will move alpha completion to 1.29.

[salehsedghpour]

@sunnylovestiramisu, to ensure we are on the same page, is there any plan to move ahead with the KEP in this release, or is it intended to be deferred to the next release which will result in marking it as Deferred to the next release?

[sunnylovestiramisu]

Deferred to the next release 1.29 is the intention @salehsedghpour

[xing-yang]

/milestone v1.29

[rayandas]

Hello @sunnylovestiramisu 👋, v1.29 Enhancements team here.

Just checking in as we approach enhancements freeze on 01:00 UTC, Friday, 6th October, 2023.

This enhancement is targeting for stage alpha for v1.29 (correct me, if otherwise)

Here's where this enhancement currently stands:

• [x] KEP readme using the latest template has been merged into the k/enhancements repo.
• [x] KEP status is marked as implementable for latest-milestone: 1.29
• [x] KEP readme has a updated detailed test plan section filled out
• [x] KEP readme has up to date graduation criteria
• [x] KEP has a production readiness review that has been completed and merged into k/enhancements.

For this KEP, we would just need to update the following:

• Update graduation criteria in the KEP README

The status of this enhancement is marked as at risk for enhancement freeze. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

[rayandas]

As the graduation criteria is updated and https://github.com/kubernetes/enhancements/pull/4245 is merged, marking this as Tracked for Enhancements Freeze. cc: @sunnylovestiramisu

[Princesso]

Hey there @sunnylovestiramisu ! 👋, v1.29 Docs team shadow here. Does this enhancement work planned for v1.29 require any new docs or modifications to existing docs? If so, please follow the steps here to open a PR against dev-1.29 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thursday, 19 October 2023. Also, take a look at Documenting for a release to get yourself familiarized with the docs requirement for the release. Thank you!

[sunnylovestiramisu]

@Princesso For 1.29 we are planning to have the API and controller change ready but the quota + admission control will happen post 1.29. It does not seem like a good idea to have doc update while not everything is ready. What is the general practice here for docs?

[msau42]

@sunnylovestiramisu I think it is fine to document what we will have. It will be marked as alpha.

[sunnylovestiramisu]

We did have a doc pr open but it got closed: https://github.com/kubernetes/website/pull/41915

[Princesso]

Thanks for the update @sunnylovestiramisu. I'll check in with the team and see what we need to do to move forward.:)

[Princesso]

Hi @sunnylovestiramisu, could you raise another PR? I checked and this one was for a blog and not docs. Kindly follow the steps here to open a PR against dev-1.29 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thursday, 19 October 2023. Also, take a look at Documenting for a release to get yourself familiarized with the docs requirement for the release. Thank you!

[sunnylovestiramisu]

@Princesso Created https://github.com/kubernetes/website/pull/43463

[sftim]

https://github.com/kubernetes/website/pull/43463 also looks like it's adding a blog article.

[kcmartin]

Hi @sunnylovestiramisu! 👋 from the v1.29 Release Team-Communications! We would like to check if you have any plans to publish a blog for this KEP regarding new features, removals, and deprecations for this release.

It seems from the comment above that this may be the case, please confirm.

If so, you need to open a PR placeholder in the website repository. The deadline will be on Tuesday 14th November 2023 (after the Docs deadline PR ready for review)

Here's the 1.29 Calendar

[rayandas]

Hey again @sunnylovestiramisu 👋, 1.29 Enhancements team here,

Just checking in as we approach code freeze at 01:00 UTC Wednesday 1st November 2023: .

Here's where this enhancement currently stands:

• [ ] All PRs to the Kubernetes repo that are related to your enhancement are linked in the above issue description (for tracking purposes).

• [ ] All PR/s are ready to be merged (they have approved and lgtm labels applied) by the code freeze deadline. This includes tests.

Please update the Issue description to include all the related PRs of this KEP under the alpha section (including https://github.com/kubernetes/kubernetes/pull/121104 I guess) in the Github Issue description. The status for this KEP is currently at risk for code freeze.

Also, please let me know if there are other PRs in k/k we should be tracking for this KEP. As always, we are here to help if any questions come up. Thanks!

[Priyankasaggu11929]

Hello @sunnylovestiramisu :wave:, 1.29 Release Team Lead here!

could you please confirm if https://github.com/kubernetes/kubernetes/pull/118863 part of code implementation PRs for this KEP?

We're currently reviewing this KEP for 1.29 Code Freeze. :slightly_smiling_face:

[carlory]

@Priyankasaggu11929 This PR https://github.com/kubernetes/kubernetes/pull/118863 is just a demo and doesn't want to be merged. API changes is https://github.com/kubernetes/kubernetes/pull/121104 and it's merged today.

[npolshakova]

@Priyankasaggu11929 This PR kubernetes/kubernetes#118863 is just a demo and doesn't want to be merged. API changes is kubernetes/kubernetes#121104 and it's merged today.

Thanks @sunnylovestiramisu!

With all the implementation(code related) PRs merged as per the issue description, this enhancement is now marked as tracked for code freeze for the 1.29 Code Freeze! 🚀

The test freeze is 01:00 UTC Wednesday 15th November 2023 / 18:00 PDT Tuesday 14th November 2023. Please make sure all test PRs are merged in by then. Let me know if there any test PRs we should track. Thanks!

[askervin]

Yet this KEP scopes cgroups as a non-goal:

OS-level IO volume attributes, for example cgroup policy

I'd like to point a cgroups configuration option in order to help the implementability of this KEP.

Modifying cgroups directly from a CSI node driver could cause conflicts. "Too many cooks in the kitchen".

However, the CSI node driver can connect to the NRI server in CRI-O or containerd. Once connected, the driver will receive pod and container lifecycle events, and it can modify all OCI parameters accordingly, including linux.resources.blockio.

When the driver gets notified on a change in VolumeAttributesClass of a volume, the driver can immediately update OCI parameters of running containers, too, through the same NRI connection.

Maybe this could be handled by a generic CSI sidecar container to help handling most common scenarios?

[salehsedghpour]

/remove-label lead-opted-in