Kubevirt fails to run on bottlerocket

[pauloalem]

Image I'm using: OS Image: Bottlerocket OS 1.24.0 (aws-k8s-1.28) Operating System: linux Architecture: amd64 Container Runtime Version: containerd://1.7.22+bottlerocket Kubelet Version: v1.28.10-eks-890c2ac Kube-Proxy Version: v1.28.10-eks-890c2ac

What I expected to happen: I am trying to setup kubevirt 1.3.1 on EKS with bottlerocket.

What actually happened: There's one kubevirt component that is currently failing to start:

virt-handler {"component":"virt-handler","level":"info","msg":"Kubevirt Seccomp profile is not enabled","pos":"virt-handler.go:521","timestamp":"2024-10-02T10:27:07.747966Z"}
virt-handler {"component":"virt-handler","level":"info","msg":"set verbosity to 2","pos":"virt-handler.go:490","timestamp":"2024-10-02T10:27:07.747976Z"}
virt-handler {"component":"virt-handler","level":"info","msg":"VSOCK server is already stopped","pos":"server.go:32","timestamp":"2024-10-02T10:27:07.747985Z"}
virt-handler Error: error relabeling file /proc/self/fd/4 with label system_u:object_r:container_file_t:s0. Reason: operation not supported
virt-handler Usage:
virt-handler   virt-chroot selinux relabel [flags]
virt-handler 
virt-handler Examples:
virt-handler virt-chroot selinux relabel <new-label> <file-path>
virt-handler 
virt-handler Flags:
virt-handler   -h, --help          help for relabel
virt-handler       --root string   safe root path which will be prepended to passed in files (default "/")
virt-handler 
virt-handler Global Flags:
virt-handler       --cpu uint       cpu time in seconds for the process
virt-handler       --memory uint    memory in bytes for the process
virt-handler       --mount string   mount namespace to use
virt-handler       --user string    switch to this targetUser to e.g. drop privileges
virt-handler 
virt-handler error relabeling file /proc/self/fd/4 with label system_u:object_r:container_file_t:s0. Reason: operation not supported
virt-handler panic: error relabeling required files: error relabeling file root: /, relative: /dev/net/tun with label system_u:object_r:container_file_t:s0. Reason: exit status 1
virt-handler 
virt-handler goroutine 1 [running]:
virt-handler main.(*virtHandlerApp).Run(0xc000682c40)
virt-handler     cmd/virt-handler/virt-handler.go:406 +0x271c
virt-handler main.main()
virt-handler     cmd/virt-handler/virt-handler.go:675 +0x49

This kubevirt component is managed by a DaemonSet. It was running with

securityContext:
  privileged: true

I've changed it to

securityContext:
  privileged: true
  seLinuxOptions:
    role: system_r
    user: system_u
    type: super_t

But it made no difference.

Kubevirt install docs mention that it requires container-selinux installed. I'm trying to figure out if bottlerocket has it, but it's unclear to me. I checked the bottlerocket node via the admin-container and can't see the container_file_t type.

seinfo -t

Types: 34
   clock_exec_t
   container_t
   runtime_exec_t
   system_t
   any_t
   api_t
   bus_t
   cache_t
   clock_t
   data_t
   etc_t
   lease_t
   os_t
   proc_t
   state_t
   mount_exec_t
   csi_exec_t
   bus_exec_t
   cni_exec_t
   api_exec_t
   network_t
   measure_t
   runtime_t
   private_t
   network_exec_t
   init_exec_t
   api_socket_t
   kernel_t
   secret_t
   init_t
   local_t
   mount_t
   super_t
   control_t

I found this PR that hints that bottlerocket does have container-selinux installed though. Or is it that bottlerocket used names that matches the ones that container-selinux uses, but are different types?

Given that the DaemonSet is running in privileged mode with super_t as type the problem then must be that Bottlerocket does not have container-selinux types installed, right? Is this case, is my only option to build my own bottlerocket image with this dependency included?

How to reproduce the problem: Install kubevirt on a bottlerocket node.

[ginglis13]

Could you please check journal logs on the node for an AVC denial to debug what process is causing this? journalctl -k | grep denial

As for the container-selinux types, the Bottlerocket selinux policies are defined here: https://github.com/bottlerocket-os/bottlerocket-core-kit/tree/develop/packages/selinux-policy. We do include container_file_t as an alias to local_t (source)

[bcressey]

super_t and privileged: true are mutually exclusive, or rather privileged: true clobbers the SELinux value and you end up running as control_t which isn't allowed to do arbitrary relabeling.

But regarding this error:

Error: error relabeling file /proc/self/fd/4 with label system_u:object_r:container_file_t:s0. Reason: operation not supported

I would not expect this to work, which is probably where the ENOTSUP error comes from - a process's file descriptors will have the same label as the process, and can't be individually relabeled.

Maybe kube-virt is getting confused and passing a file descriptor path instead of a file path for some reason?

[pauloalem]

@ginglis13 Unfortunately there does not seem like there's anything in the journal logs:

[root@admin]# sheltie journalctl -k | grep -i denied

@bcressey I've tried untoggling privileged but that results in the pod not starting at all.

Forbidden: Bidirectional mount propagation is available only to privileged containers

Maybe there is there a set of security settings that are equivalent to privileged that I could use in conjunction with the super_t label?

This is where the relabel is being done: https://github.com/kubevirt/kubevirt/blob/03c48fca9dcb9b1c27518fed20faa9648c04acb5/cmd/virt-handler/virt-handler.go#L401

I don't know why the error mentions /proc/self/fd/4 instead of /dev/net/tun but if you look further down:

virt-handler error relabeling file /proc/self/fd/4 with label system_u:object_r:container_file_t:s0. Reason: operation not supported
virt-handler panic: error relabeling required files: error relabeling file root: /, relative: /dev/net/tun with label system_u:object_r:container_file_t:s0. Reason: exit status 1