Setting to control bottlerocket host cgroup cpu allocation

[james-masson]

What I'd like:

A setting to control the CPUs allowed to run the internal cgroup for the bottlerocket OS/containers itself.

eg.

[ settings.os-resources.cgroup ]
"allowed-cpus" = "0-3"

We run workloads with k8s cpu-manager policy "static", and Pods in the "Guarenteed" QoS class. We want to dedicate entire CPU cores (and NUMA nodes) to workloads, and not have a context-switch if the OS decides to schedule a bottlerocket-internal process on a core. Hence the need to have the cgroup containing the OS/containers on a known set of CPUs, on a known NUMA node.

Kubernetes should be free to schedule on it's own cgroup, which has the CPUs that are not claimed by bottlerocket's cgroup.

Any alternatives you've considered:

Building and supporting a custom variant with this built-in as default.

[gthao313]

@james-masson Thanks for opening this issue! We will investigate on it. : )

[james-masson]

I raised a question about the best way to achieve this for now.

https://github.com/bottlerocket-os/bottlerocket/discussions/3889

What I needs is a way to be able to set permanent systemd params.

Add AllowedCPUs to kubepods.slice runtime.slice system.slice and user.slice to restrict what CPUs they can use.