bottlerocket-os
Enable low-jitter kernel options for kernel-6.1
Issue number:
From private discussions.
Description of changes:
Enabling a selection of Kernel build-options to allow users to select various low-latency and low-jitter behaviours from the kernel.
These kernel options are fairly typical, you'll find them in Redhat and Ubuntu distro kernels.
None of these options do anything without the user opting into them via kernel parameters - default behaviour for your users will stay the same.
Testing done:
Local build and deploy into our AWS accounts, benchmarking using these kernel options.
Terms of contribution:
By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.
None of these options do anything without the user opting into them via kernel parameters - default behaviour for your users will stay the same.
We'll need to run this by the Amazon Linux kernel engineers. Ideally the change would happen upstream in the AL kernels, or at least we'd have some indication that they're likely to be picked up there. It's also not necessarily what the default is that matters: if it's possible to enable something, and people do, ideally that wouldn't push them into an unsupported state.
I took a look at this and we already have a few of these enabled in the 6.1 kernel, here are the ones that are not:
CONFIG_RCU_NOCB_CPU=y isn't set.
CONFIG_CONTEXT_TRACKING_USER=y isn't set but CONFIG_HAVE_CONTEXT_TRACKING_USER=y and CONFIG_CONTEXT_TRACKING=y and CONFIG_CONTEXT_TRACKING_IDLE=y are set.
CONFIG_NO_HZ_FULL=y is not set but CONFIG_NO_HZ_IDLE=y is set.
The rest already exist. I'll follow up when I have some answers as @bcressey mentioned I'd like to see if we can include these in Amazon Linux as well so these settings are consistent for both. Thanks for bringing this up @james-masson!
CONFIG_RCU_NOCB_CPU=y isn't set.
CONFIG_NO_HZ_FULL=y is not set
These are the two important ones. They enable the functionality we need.
Updated the PR to include just the specific modules to enable, not the full config.
Dumb question @james-masson , will it be possible to make them m instead of y?
Dumb question @james-masson , will it be possible to make them
minstead ofy?
No, it's a boolean.
https://github.com/torvalds/linux/blob/e0cce98fe279b64f4a7d81b7f5c3a23d80b92fbc/kernel/time/Kconfig#L124
It's not a self-contained bit of functionality like a device driver. It's a switch that enables alternative behaviour in the scheduler and IRQ handling and similar.
Dumb question @james-masson , will it be possible to make them
minstead ofy?
It's a switch that enables alternative behaviour in the scheduler and IRQ handling and similar.
I perhaps wasn't clear enough here...
Enabling this kernel build config does not change any default behaviour in the kernel AFAIK. It just enables a capability in the kernel.
Only when you set rcu_nocbs=<insert-cpu-list-here> as a kernel parameter does the behaviour change, and that's something you always want to leave to the user.
It's a similar situation with the nohz_full kernel parameter. The kernel build config just enables the capability, user opts-in to the capability with a kernel parameter.
I was able to sync with some Amazon Linux engineers and they found some interesting data here. It looks like our aarch64 kernels already have this enabled. I confirmed this with a node I had running:
# After dumping the config from /proc/config.gz to the disk for decompression:
[root@admin]# grep NO_HZ_FULL /.bottlerocket/rootfs/var/log/config
CONFIG_NO_HZ_FULL=y
[root@admin]# grep RCU_NOCB_CPU /.bottlerocket/rootfs/var/log/config
CONFIG_RCU_NOCB_CPU=y
# CONFIG_RCU_NOCB_CPU_DEFAULT_ALL is not set
They also linked me to https://elixir.bootlin.com/linux/v6.1.92/source/kernel/time/Kconfig#L145 which, at least for NO_HZ_FULL has pretty straightforward instructions to enable this.
Since these are already enabled for aarch64, I think that is a good sign that we could enable this on x86_64 without adverse affect. It does appear that these options don't trigger performance changes until provided the matching boot configuration on the command line.
A quick update on this change. It looks like Amazon Linux will be taking this change in their configuration so we should see this come through on a config change in an upcoming release of their kernel. We will watch for that to make sure it happens but we shouldn't need this PR once that change happens. I'll close out this PR once I see that change come through. Thanks for opening this PR!
https://github.com/bottlerocket-os/bottlerocket-core-kit/pull/29 Will take these changes making this PR no longer needed. Thanks for bringing this to our attention! Once that PR is merged, the x86_64 and aarch64 kernels will have these options configured.
I'm going to close this in favor of https://github.com/bottlerocket-os/bottlerocket-core-kit/pull/29