system-validators icon indicating copy to clipboard operation
system-validators copied to clipboard

Published 20 hours ago • verified publisher icon kubernetes

Update the kernel configuration for cgroup v2

Open KentaTada opened this issue 2 years ago • 18 comments
trafficstars

Kubernetes 1.25 brings cgroup v2 to GA.  cgroup v2 needs some additional kernel configs. For example, you need to enable CONFIG_CGROUP_BPF if you want to use the the device controller. When it comes to Kubernetes, I have never investigated what config is actually needed. But I just create an issue at first.

KentaTada avatar Mar 28 '23 11:03 KentaTada

https://github.com/kubernetes/system-validators/pull/12#issuecomment-607668321 by @odinuge

CONFIG_CGROUP_BPF - Required for cgroupv2 (for controlling devices)

https://github.com/containerd/containerd/pull/3799#issuecomment-555740694 by @AkihiroSuda

kernel >= 4.15 with CONFIG_CGROUP_DEVICE and CONFIG_CGROUP_BPF is required.

/cc @bobbypage @mrunalp for cgroup v2 GA

https://github.com/kubernetes/minikube/pull/6572

pacoxu avatar Mar 29 '23 03:03 pacoxu

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle stale
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot avatar Jun 27 '23 04:06 k8s-triage-robot

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle rotten
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot avatar Jan 19 '24 03:01 k8s-triage-robot

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Reopen this issue with /reopen
  • Mark this issue as fresh with /remove-lifecycle rotten
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

k8s-triage-robot avatar Feb 18 '24 03:02 k8s-triage-robot

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Reopen this issue with /reopen
  • Mark this issue as fresh with /remove-lifecycle rotten
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot avatar Feb 18 '24 03:02 k8s-ci-robot

/remove-lifecycle rotten @pacoxu @KentaTada do we still need such a change?

neolit123 avatar Feb 18 '24 09:02 neolit123

Yes. We need to investigate and prepare for the kernel configuration for cgroupv2. If I have time, I'll investigate it.

KentaTada avatar Mar 04 '24 14:03 KentaTada

thanks, @KentaTada

neolit123 avatar Mar 04 '24 14:03 neolit123

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle stale
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot avatar Jun 02 '24 15:06 k8s-triage-robot

/remove-lifecycle stale

KentaTada avatar Jun 04 '24 00:06 KentaTada

The v1.31 KEP https://github.com/kubernetes/enhancements/issues/4569

  • https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/4569-cgroup-v1-maintenance-mode/README.md KEP-4569: Moving cgroup v1 support into maintenance mode

I opened https://github.com/kubernetes/system-validators/pull/37 for the kernel version.

  • for KernelConfig part, I need to check the list.

pacoxu avatar Jul 24 '24 10:07 pacoxu

Yes. We need to investigate and prepare for the kernel configuration for cgroupv2. If I have time, I'll investigate it.

@KentaTada should we include this in the next system-validators release too?

@pacoxu recently added:

  • https://github.com/kubernetes/system-validators/pull/38
  • https://github.com/kubernetes/system-validators/pull/37

neolit123 avatar Aug 12 '24 06:08 neolit123

@KentaTada should we include this in the next system-validators release too?

Yes. Although I haven't completely caught up with these commits yet, we should include the info about cgroup v2 before moving cgroup v1 support into maintenance mode.

BTW, we also need to update the list of kernel config. For example, CONFIG_CGROUP_FREEZER is not actually required for cgroup v2. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/init/Kconfig?h=v5.8#n1006 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/init/Kconfig?h=v6.10#n1103

KentaTada avatar Aug 12 '24 11:08 KentaTada

would you have time to send a PR or give us the info of all the required changes that we need to do?

neolit123 avatar Aug 12 '24 13:08 neolit123

we also need to update the list of kernel config. For example, CONFIG_CGROUP_FREEZER is not actually required for cgroup v2. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/init/Kconfig?h=v5.8#n1006 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/init/Kconfig?h=v6.10#n1103

For cgroup v2, we use cgroup.freeze instead, which needs kernel 5.2+.

pacoxu avatar Aug 13 '24 03:08 pacoxu

I'm sorry but  I need to support my family this week. In addition to that, I need to chair a meeting of KubeDay and my local CNCF-related eBPF event in August. https://sched.co/1eh9w https://community.cncf.io/e/m4e8cg/

For cgroup v2, we use cgroup.freeze instead, which needs kernel 5.2+.

What I wanted to say is that we need to prepare for how to detect what cgroup v2 features are available. Unlike in v1, the kernel config alone cannot determine whether the required v2 features are enabled. Maybe, it is possible if this validator checks for the existence of the file of cgroup.freeze. We also need to think about BPF-based interfaces like the device controller. In addition to that, we need to confirm what v2 features are currently needed for k8s at first. This change should be made with caution because k8s users all over the world may recompile their kernel by the result of kubeadm.

KentaTada avatar Aug 14 '24 01:08 KentaTada

I'm sorry but I need to support my family this week. In addition to that, I need to chair a meeting of KubeDay and my local CNCF-related eBPF event in August.

1.31 was just released so this is planned for 1.32. we have a whole k8s release cycle to tackle the feature detection for cgroups v2.

What I wanted to say is that we need to prepare for how to detect what cgroup v2 features are available. Unlike in v1, the kernel config alone cannot determine whether the required v2 features are enabled. Maybe, it is possible if this validator checks for the existence of the file of cgroup.freeze. We also need to think about BPF-based interfaces like the device controller.

i think having the cgroups v2 validation load additional files should be fine.

In addition to that, we need to confirm what v2 features are currently needed for k8s at first.

@pacoxu maybe we need to ask SIG node and bring them this this thread?

This change should be made with caution because k8s users all over the world may recompile their kernel by the result of kubeadm.

completely agree.

neolit123 avatar Aug 14 '24 06:08 neolit123

What I wanted to say is that we need to prepare for how to detect what cgroup v2 features are available. Unlike in v1, the kernel config alone cannot determine whether the required v2 features are enabled. Maybe, it is possible if this validator checks for the existence of the file of cgroup.freeze. We also need to think about BPF-based interfaces like the device controller.

i think having the cgroups v2 validation load additional files should be fine.

In https://github.com/kubernetes/kubernetes/pull/126595, we may use cpu.stat check instead of version check there.

pacoxu avatar Aug 14 '24 09:08 pacoxu

@KentaTada @pacoxu can this be done for the 1.32 release cycle?

core freeze is 7th of Nov 2024. https://github.com/kubernetes/sig-release/tree/master/releases/release-1.32

neolit123 avatar Sep 24 '24 09:09 neolit123

I would like to give it a try in this release.

pacoxu avatar Sep 24 '24 10:09 pacoxu