Trying to modprobe allowed true modules does not work

[chris-milsted]

Image I'm using:

$ cat /etc/bottlerocket-release NAME=Bottlerocket ID=bottlerocket VERSION="1.9.0 (aws-k8s-1.22)" PRETTY_NAME="Bottlerocket OS 1.9.0 (aws-k8s-1.22)" VARIANT_ID=aws-k8s-1.22 VERSION_ID=1.9.0 BUILD_ID=159e4ced HOME_URL="https://github.com/bottlerocket-os/bottlerocket" SUPPORT_URL="https://github.com/bottlerocket-os/bottlerocket/discussions" BUG_REPORT_URL="https://github.com/bottlerocket-os/bottlerocket/issues"

What I expected to happen:

I expect the additional modules to be loaded using standard linux workflow, e.g. modprobe.

I am trying to load some additional modules as a follow up to https://github.com/bottlerocket-os/bottlerocket/pull/1953. I have created an eksctl file which is as follows:

---
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: chris-bottlerocket-cluster
  region: eu-west-2

nodeGroups:
  - name: ng1-public
    instanceType: m5.xlarge
    desiredCapacity: 1
    amiFamily: Bottlerocket
    labels:
      "owner": "chris"
    bottlerocket:
      enableAdminContainer: true
      settings:
        motd: "Hello, eksctl!"
        kernel:
          lockdown: none
          modules:
            target_core_mod:
              allowed: true
            target_core_user:
              allowed: true
            tcm_loop:
              allowed: true
            uio:
              allowed: true     
            configfs:
              allowed: true  
    ssh:
      # Enable ssh access (via the admin container)
      allow: true
      publicKeyName: <name>

This was following the documentation here:

https://github.com/bottlerocket-os/bottlerocket#kernel-settings

I expected to be able to ssh into the machine and now modprobe to load modules.

What actually happened:

[ec2-user@admin]$ modprobe target_core_mod
modprobe: ERROR: could not insert 'target_core_mod': Operation not permitted

How to reproduce the problem:

See above, use the eksctl command and ssh into the machine.

[bcressey]

This looks like fallout from #2296 - what's happening is that modprobe in the admin container doesn't recognize zstd-compressed modules, and passes the compressed module directly to the kernel. If lockdown is enabled, this compressed blob is treated as an unsigned module and rejected; if it's disabled, validation of the ELF structure of the module will fail.

As a workaround you can use the host's modprobe via:

sudo sheltie modprobe target_core_mod

I've opened https://github.com/bottlerocket-os/bottlerocket-admin-container/issues/66 to track the fix in the admin container.

[chris-milsted]

Hi Ben,

Just to confirm, the fix you are investigating will work from any container (suitably privileged) and not just need to use the admin container? i.e. to install components like CSI plugins, the node plugin daemon set may need to load whitelisted modules.

https://kubernetes-csi.github.io/docs/deploying.html

Thanks

Chris

[bcressey]

Just to confirm, the fix you are investigating will work from any container (suitably privileged) and not just need to use the admin container?

That fix is specific to the admin container. Whether other containers need a fix will depend on whether they are running a base distro where kmod is built with zstd support. I'm not sure how widespread that support is. Newer versions of Alpine, Fedora, and Ubuntu would have it.

There are a few workarounds that wouldn't require updating the container's base image - using an init container in the deployment to load modules, or a bootstrap container to load them when Bottlerocket boots, or using the host's modprobe - but ideally Bottlerocket would be compatible with these solutions out of the box and not require a workaround.

My current thinking is that the right path is to partially revert #2296 and use xz instead of zstd compression, to broaden the base of Linux distros that just work. It'd help to hear what distro you're using and whether kmod supports xz, to avoid replacing one breaking change with another.

This can be checked by running kmod -V:

/ # kmod -V
kmod version 29
+ZSTD +XZ +ZLIB +LIBCRYPTO -EXPERIMENTAL

[chris-milsted]

# kmod -V
kmod version 25
+XZ +ZLIB +OPENSSL -EXPERIMENTAL

Using a RHEL8 UBI as base image.

[markusboehme]

Hi Chris,

The recently released Bottlerocket v1.9.1 switched to the xz format for kernel module compression which, at least for now, is more widely supported than zstd. Please give it a go and let me know if you hit any problems!

Thanks Markus