Split metal and non-metal kernel config

[zmrow]

As time goes on and we continue to add kernel modules to support new and different hardware, we don't want our non-metal variants to carry all of the extra modules. We should split out the metal-specific config.

There are a few considerations here however:

• We don't want a separate kernel package to maintain
• We need to ensure that the kernel gets re-built for metal/non-metal as required (will need to use the variant-sensitive key in Cargo.toml)

[webern]

will need to use the variant-sensitive key in Cargo.toml

We should consider some additional logic to ensure that the kernel is only rebuilt when switching between metal and other versions, and not rebuilt when switching between eg aws-ecs and aws-k8s

[foersleo]

Yes, I think there is some steps to this here. I am thinking of something like the following:

• [x] Separate the config but have the same build with all options (Done with https://github.com/bottlerocket-os/bottlerocket/pull/2264)
• [x] Separate the builds to only have the metal specific configs build into the metal variants
• [ ] build time improvements *

(*) For the build time improvements there is multiple levels to that I think:

1. Re-use builds as much as possible (as mentioned by Matt above) - This will probably be some work on the capabilities of the build system
2. What parts of one build might we be able to re-use through something like ccache or similar. - This will need some experimentation on how such caching tools work with the kernel and changing configs.

[foersleo]

I have been looking into how to split the builds.

Doing this the basic way of just adding some logic in the kernel.spec that determines when to merge the metal config and adding the variant-sensitive flag to the kernel package is straight forward and works in some testing I have done. However it also incurs some extra package rebuilds when switching variants within the metal platform or outside the metal platform.

So probably we want to think about adding some feature to specify a specific part of the variant (platform, runtime, family, ...) and a specific value for that part we want to have rebuilds of a specific package for. I have done some poking around, but I probably need to get more familiar with buildsys to understand how to implement that.

[foersleo]

The functional parts of splitting the configs and building the kernel with the variant appropriate config are done. As discussed in https://github.com/bottlerocket-os/bottlerocket/pull/2279 the impact of re-building the kernel with each variant is less than expected as the variant specific build of the os package takes similar amounts of time as the kernel build.

I will either way have a look on how we might be able to recoup some of the additional time spent in building with this split.

[bcressey]

It might be worth expanding on the concepts in bottlerocket-variant and adding more fine-grained sensitivity in buildsys.

Today we have variant-sensitive but we could do one of these instead:

• add a platform-sensitive flag, so we can express that only changes from metal to aws or vice-versa should trigger a rebuild
• use a more complex deserialized type for variant-sensitive, where a boolean would cover any change, and variant-sensitive = "platform" would only trigger if the platform part changed