kata + privileged: `failed to create shim task: QMP command failed: The device is not writable: Permission denied: unknown`

[AkihiroSuda]

Description

$ sudo nerdctl run -it --rm --runtime=io.containerd.kata.v2 --privileged alpine
FATA[0004] failed to create shim task: QMP command failed: The device is not writable: Permission denied: unknown

Steps to reproduce the issue

1. sudo PATH=/opt/kata/bin:$PATH containerd (as root)
2. sudo nerdctl run -it --rm --runtime=io.containerd.kata.v2 --privileged alpine

Describe the results you received and expected

Received: failed with the error shown above Expected: should work

Non-privileged mode works as expected

What version of nerdctl are you using?

$ sudo nerdctl version
Client:
 Version:       v0.22.2
 OS/Arch:       linux/amd64
 Git commit:    2899222cb0715f1e5ffe356d10c3439ee8ee3ba4
 buildctl:
  Version:      v0.10.0-380-g874eef9b
  GitCommit:    874eef9b70dbaf4f074d2bc8f4dc64237f8e83a0

Server:
 containerd:
  Version:      v1.6.6
  GitCommit:    10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
 runc:
  Version:      1.1.2
  GitCommit:    v1.1.2-0-ga916309f

$ sudo /opt/kata/bin/kata-runtime -v
kata-runtime  : 2.5.0-rc0
   commit   : ac91fb7a126238b595afa59c17fbb82a12c4f5ad
   OCI specs: 1.0.2-dev

Are you using a variant of nerdctl? (e.g., Rancher Desktop)

No response

Host information

$ sudo nerdctl info
Client:
 Namespace:     default
 Debug Mode:    false

Server:
 Server Version: v1.6.6
 Storage Driver: overlayfs
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Log: fluentd journald json-file
  Storage: native overlayfs
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.15.0-43-generic
 Operating System: Ubuntu 22.04 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 15.59GiB
 Name: suda-ws01
 ID: e4b49f8a-ed36-4fab-98a2-bc5d0e51d2ac

[AkihiroSuda]

Potentially related:

• https://github.com/kata-containers/tests/issues/3002#issuecomment-1049811275

cc @liubin

[liubin]

Thank you, @AkihiroSuda, there are also some issues related to this:

• https://github.com/kata-containers/kata-containers/issues/4488
• https://github.com/kata-containers/kata-containers/issues/3968

One approach is to add a command line option, for example named no-host-devices, which like privileged_without_host_devices of containerd's config.toml.

diff --git a/cmd/nerdctl/run.go b/cmd/nerdctl/run.go
index 8729953..726060d 100644
--- a/cmd/nerdctl/run.go
+++ b/cmd/nerdctl/run.go
@@ -196,6 +196,7 @@ func setCreateFlags(cmd *cobra.Command) {
        cmd.Flags().StringSlice("cap-drop", []string{}, "Drop Linux capabilities")
        cmd.RegisterFlagCompletionFunc("cap-drop", capShellComplete)
        cmd.Flags().Bool("privileged", false, "Give extended privileges to this container")
+       cmd.Flags().Bool("no-host-devices", false, "Don't pass host devices to privileged containers")
        // #endregion
 
        // #region runtime flags
diff --git a/cmd/nerdctl/run_linux.go b/cmd/nerdctl/run_linux.go
index 106a11b..399156b 100644
--- a/cmd/nerdctl/run_linux.go
+++ b/cmd/nerdctl/run_linux.go
@@ -108,8 +108,17 @@ func setPlatformOptions(opts []oci.SpecOpts, cmd *cobra.Command, id string) ([]o
        if err != nil {
                return nil, err
        }
+
        if privileged {
                opts = append(opts, privilegedOpts...)
+
+               noHostDevices, err := cmd.Flags().GetBool("no-host-devices")
+               if err != nil {
+                       return nil, err
+               }
+               if !noHostDevices {
+                       opts = append(opts, oci.WithHostDevices)
+               }
        }
 
        b4nnOpts, err := bypass4netnsutil.GenerateBypass4netnsOpts(securityOptsMaps, labelsMap, id)
diff --git a/cmd/nerdctl/run_security_linux.go b/cmd/nerdctl/run_security_linux.go
index 8163ad2..7be9a88 100644
--- a/cmd/nerdctl/run_security_linux.go
+++ b/cmd/nerdctl/run_security_linux.go
@@ -36,7 +36,6 @@ import (
 var privilegedOpts = []oci.SpecOpts{
        oci.WithPrivileged,
        oci.WithAllDevicesAllowed,
-       oci.WithHostDevices,
        oci.WithNewPrivileges,
 }

If users want to pass devices to container, they can use --device option.

[AkihiroSuda]

@liubin Thanks, SGTM, but I guess the flag name should be like --privileged-without-host-devices or --security-opt privileged-without-host-devices

[liubin]

@AkihiroSuda thank you, I created #1291 to fix this.