Node memory swap support

[ehashman]

Enhancement Description

• One-line enhancement description (can be used as a release note): Kubernetes nodes support swap memory.
• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2400-node-swap
• Discussion Link: https://docs.google.com/document/d/1CZtRtC8W8FwW_VWQLKP9DW_2H-hcLBcH9KBbukie67M/edit# Kubelet/Kubernetes should work with Swap Enabled kubernetes#53533
• Primary contact (assignee): @iholder101 @kannon92 @pacoxu
• Responsible SIGs: Node
• Enhancement target (which target equals to which milestone):
  • Alpha release target (x.y): 1.22
  • Beta1 release target (x.y): 1.28
  • Beta2 release target (x.y): 1.29-1.31
  • Stable release target (x.y): 1.32
• [x] Alpha
  • [x] KEP (k/enhancements) update PR(s): https://github.com/kubernetes/enhancements/pull/2602
  • [x] Code (k/k) update PR(s):
    • [x] CI/test infra changes: https://github.com/kubernetes/test-infra/pull/22602 https://github.com/kubernetes/test-infra/pull/22809
    • [x] CRI and kubelet changes https://github.com/kubernetes/kubernetes/pull/102823
  • [x] Docs (k/website) update PR(s): https://github.com/kubernetes/website/pull/28838
• [x] Beta1
  • [x] KEP (k/enhancements) update PR(s):
    • [x] https://github.com/kubernetes/enhancements/pull/2858
    • [x] https://github.com/kubernetes/enhancements/pull/3957
  • [x] Issues and Code (k/k) update PR(s):
    • Support cgroups v2 with auto-calculated LimitedSwap for Burstable QoS pods
      • [x] https://github.com/kubernetes/kubernetes/issues/119427
      • [x] https://github.com/kubernetes/kubernetes/issues/119428
      • [x] https://github.com/kubernetes/kubernetes/pull/118764
    • [x] Add Swap stats to /stats/summary and /metrics/resource
      • [x] https://github.com/kubernetes/kubernetes/issues/119425
      • [x] https://github.com/kubernetes/kubernetes/pull/118865
    • [x] Determine a set of metrics for measuring node performance with swap enabled
      • [x] https://github.com/kubernetes/kubernetes/issues/119424
      • [x] https://github.com/kubernetes/kubernetes/pull/118865
      • [x] https://github.com/google/cadvisor/pull/3339
    • [x] Improve test coverage and make sure tests are green
      • [x] https://github.com/kubernetes/kubernetes/issues/119430
      • [x] https://github.com/kubernetes/kubernetes/pull/118764
  • [x] Docs (k/website) update(s):
    • [x] https://github.com/kubernetes/website/pull/42009
    • [X] https://github.com/kubernetes/website/pull/42167
  • [x] Misc
    • [x] https://github.com/google/cadvisor/pull/3293
    • [x] https://github.com/kubernetes/kubernetes/pull/119225
  • Archive / Outdated
• [x] Beta2
  • [x] KEP (k/enhancements) update PR(s):
    • [x] 1.29 https://github.com/kubernetes/enhancements/pull/4275
    • [x] 1.30 https://github.com/kubernetes/enhancements/pull/4401
    • [x] 1.30 https://github.com/kubernetes/enhancements/pull/4484
  • [x] Issues and Code (k/k) update PR(s):
    • ~~https://github.com/kubernetes/kubernetes/pull/122175~~
    • ~~https://github.com/kubernetes/kubernetes/pull/122241~~
    • [x] 1.30
      • [x] https://github.com/kubernetes/kubernetes/pull/122745
      • [x] https://github.com/kubernetes/kubernetes/pull/123738
    • [x] 1.31
      • [x] https://github.com/kubernetes/kubernetes/pull/123557
      • [x] https://github.com/kubernetes/kubernetes/pull/123749
      • [x] 1.31 https://github.com/kubernetes/kubernetes/pull/124060
      • [x] 1.31 https://github.com/kubernetes/kubernetes/pull/125277
      • [x] 1.31 https://github.com/kubernetes/kubernetes/pull/125402
  • [x] Docs (k/website) update(s):
    • [x] 1.30 https://github.com/kubernetes/website/pull/45128
• [ ] GA (related PRs, can be assigned to a specific track later)
  • [ ] KEP (k/enhancements) update PR(s):
    • [ ] https://github.com/kubernetes/enhancements/pull/4701
  • [ ] Issues and Code (k/k) update PR(s):
    • [x] https://github.com/kubernetes/kubernetes/issues/120801
    • [ ] https://github.com/kubernetes/kubernetes/issues/120800
    • [x] https://github.com/kubernetes/kubernetes/issues/120798.
    • [x] https://github.com/kubernetes/kubernetes/issues/105978
    • [ ] https://github.com/kubernetes/kubernetes/issues/121820
    • Add the ability to set a system-reserved quantity of swap on a node
      • [ ] https://github.com/kubernetes/kubernetes/issues/120802
      • [x] ~~https://github.com/kubernetes/kubernetes/issues/105019~~ dup by 120802
      • ~~https://github.com/kubernetes/kubernetes/pull/105271~~
    • ~~https://github.com/kubernetes/kubernetes/pull/119327~~
    • [x] https://github.com/kubernetes/kubernetes/issues/122279
  • [ ] Docs (k/website) update(s):

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

/sig node

[karan]

@cookieisaac

[ehashman]

/stage alpha /milestone v1.22

[jrsapi]

Greetings @ehashman! Enhancement shadow checking in with a few reminders. 1.22 Enhancements Freeze starts at 23:59:59 pst on Thursday, May 13. A few items needing review for this KEP:

• Enhancement README

• PRR approved

• https://github.com/kubernetes/enhancements/issues/2400 updated with current target and alpha information.

Thanks!

[ehashman]

All of this is covered in https://github.com/kubernetes/enhancements/pull/2602

[jrsapi]

Greetings @ehashman! Thanks for the follow-up. After reviewing the KEP and PRR everything looks on target. The enhancement is marked at risk but once merged we can move its status to tracked. A reminder that tomorrow 5/13 the Enhancement freeze starts at 23:59:59 PST.

Thanks!

[ehashman]

Work breakdown for 1.22

• Documentation for enabling swap (TBD - @ehashman?)
• CI environment/test updates (@ike-ma)
  • Build images with swap for 2 Linux distros
  • Add jobs to test-infra that use the images and enable the swap feature flag/kubelet option for node e2e suite
  • See also Test Plan
• CRI and kubelet changes (@ehashman)
  • Complete API changes per KEP
  • Add e2e tests as appropriate

External to k8s but still need to happen:

• Containerd update to use new CRI
• CRI-O update to use new CRI

Once the above CRI updates happen, ensure that CI environment is using latest container runtimes with updated CRI.

[jrsapi]

Greetings @ehashman , Enhancement shadow checking with a reminder that we are 2 weeks away from code freeze (July 8, 2021). Can you like the k/k PR/s that are needed to implement this enhancement for the 1.22 milestone?

[jrsapi]

Greetings @ehashman, A friendly reminder that code freeze is this Thursday, July 8th and we're tracking the following k/k PR:

• https://github.com/kubernetes/kubernetes/pull/102823 The PR has to be merged by code freeze and currently, this KEP is marked "At Risk". Also, Docs PR placeholder deadline is Friday, July 9th. Here are instructions on opening a PR in dev-1.22 branch.

Thanks!

[ehashman]

PR just merged. Docs placeholder is https://github.com/kubernetes/website/pull/28838

[jrsapi]

Awesome! Thanks for the update. Moving this to "Tracked".

[ehashman]

We are good to go for 1.22! Docs complete.

[ehashman]

/milestone v1.23

[ehashman]

/stage beta

[kevindelgado]

Hi @ehashman 1.23 Enhancements team here. Just checking in as we approach enhancements freeze on Thursday 09/09. Here's where this enhancement currently stands:

• [X] KEP file using the latest template has been merged into the k/enhancements repo.
• [X] KEP status is marked as implementable
• [X] KEP has a test plan section filled out.
• [X] KEP has up to date graduation criteria. ~(Graduation Criteria for beta currently says "tentative")~
• [X] KEP has a production readiness review that has been completed and merged into k/enhancements.

~Looks like for this one we would just need the KEP/PRR to merge (and the "tentative" label on graduation criteria removed) by enhancements freeze :)~

Good to go now!

Thanks!

[kevindelgado]

Hi @ehashman

Ping! As a reminder your PR (#2858) needs to merge by EOD PST tomorrow September 9th to be included in the 1.23 Release. After that time you will need to request an exception.

Lmk if you need anything, Kevin

[ehashman]

Hello @kevindelgado,

PRR is now approved and the "tentative" bit has been moved to GA. Awaiting final node approval.

[jlbutler]

Hi @ehasman :wave: 1.23 Docs lead here.

This enhancement is marked as 'Needs Docs' for the 1.23 release.

Please open a PR against the dev-1.23 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thu November 18, 11:59 PM PDT.

Thanks!

[SergeyKanzhelev]

As discussed at SIG Node meeting on October 26, 2021 we are removing this enhancement from 1.23 milestone. @kubernetes/release-managers can you please remove the milestone label?

[endocrimes]

Swap support has now landed in containerd: https://github.com/containerd/containerd/pull/6320 - Have a branch prepped for when a release drops to include in test-infra.

[ehashman]

/milestone v1.24

[yangjunmyfm192085]

/cc

[gracenng]

Hi @ehashman ! 1.24 Enhancements team here. Just checking in as we approach enhancements freeze on 18:00pm PT on Thursday Feb 3rd. This enhancements is targeting beta for 1.24.

The status of this enhancement is track as tracked , no action is required for Enhancements Freeze. Thanks!

[nate-double-u]

Hi @ehashman :wave: 1.24 Docs lead here.

This enhancement is marked as Needs Docs for the 1.24 release.

Please follow the steps detailed in the documentation to open a PR against the dev-1.24 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thursday, March 31st, 2022 @ 18:00 PDT.

Also, if needed take a look at Documenting for a release to familiarize yourself with the docs requirement for the release.

Thanks!

[ehashman]

Work for the 1.24 release was organized on a project board: https://github.com/orgs/kubernetes/projects/69

We also have a slack channel: #sig-node-swap

I am doubtful this will make the soft node freeze on Mar. 4 (this Friday) given how much work is left to complete.

[ehashman]

/milestone clear per https://docs.google.com/document/d/1Ne57gvidMEWXR70OxxnRkYquAoMpt56o75oZtg-OeBg/edit#bookmark=id.w24ewvhebv4u

[gracenng]

Removing this enhancements from the release

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[pacoxu]

/remove-lifecycle stale

[marosset]

/milestone v1.26 /label lead-opted-in (I'm doing this on behalf of @ruiwen-zhao / SIG-node)

[rhockenbury]

Hello @ehashman 👋, 1.26 Enhancements team here.

Just checking in as we approach enhancements freeze on 18:00 PDT on Thursday 6th October 2022.

This enhancement is targeting for stage alpha for 1.26 (from this comment, we'll track it as a major change for alpha, rather than beta graduation)

Here's where this enhancement currently stands:

• [ ] KEP readme using the latest template has been merged into the k/enhancements repo.
• [ ] KEP status is marked as implementable for latest-milestone: 1.26
• [ ] KEP readme has a updated detailed test plan section filled out
• [ ] KEP readme has up to date graduation criteria
• [X] KEP has a production readiness review that has been completed and merged into k/enhancements.

For this KEP, we would just need to update the following:

• Update latest-milestone, stage and milestone in the KEP.yaml
• Update the test plan section to match the latest template
• Merge up https://github.com/kubernetes/enhancements/pull/3571

The status of this enhancement is marked as at risk. Please keep the issue description up-to-date with appropriate stages as well. Thank you!