Running sysbox on an initrd device does not work - jailing process inside rootfs caused: permission denied: unknown

[yogch]

Hi, I am trying to run sysbox inside a physical device. It has kernel 6.1.107 which was compiled by me. I've added all the necessary kernel modules in the kernel's x86_64-all.config.

• iptables/nf/netfilter
• bridge
• bpf
• fuse/cuse/virtio
• user ns
• ...

I've changed the /etc/docker/daemon.json so the data-root is on a mounted disk - not the initrd file system (docker yells about it). Afterwards, docker works fine (running docker run --rm -it hello-world works). Also, I had to add Environment="DOCKER_RAMDISK=true" to docker's systemd file:

# /etc/systemd/system/docker.service.d/override.conf
[Service]
Environment="DOCKER_RAMDISK=true"

Afterwards I've tried to install sysbox properly according to this site's installation guide (installing the .deb). Same as docker, I've changed /lib/systemd/system/sysbox-mgr.service to have a --data-root on the mounted disk. Also /lib/systemd/system/sysbox-fs.service with --mountpoint.

Then, when trying to run: docker run --rm -it --runtime=sysbox-runc hello-world I get:

docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: container_linux.go:439: starting container process caused: process_linux.go:608: container init caused: rootfs_linux.go:119: jailing process inside rootfs caused: permission denied: unknown

Please let me know - is there a missing kernel module? Do I miss some configuration? Thanks.

~# sysbox-mgr --version
sysbox-mgr
        edition:        Community Edition (CE)
        version:        0.6.7
        commit:         aaeff6c5dc70c137e62166474a309ca5fc42d044
        built at:       Fri May  9 21:15:06 UTC 2025
        built by:       Cesar Talledo

~# docker --version
Docker version 28.3.0, build 38b7060

~# cat /etc/docker/daemon.json
{
    "data-root": "/mnt/mymount/var/lib/docker",
    "runtimes": {
        "sysbox-runc": {
            "path": "/usr/bin/sysbox-runc"
        }
    },
    "bip": "172.20.0.1/16",
    "default-address-pools": [
        {
            "base": "172.25.0.0/16",
            "size": 24
        }
    ]
}

~# cat /lib/systemd/system/sysbox-mgr.service
...
ExecStart=/usr/bin/sysbox-mgr --data-root /mnt/mymount/var/lib/sysbox
...

~# cat /lib/systemd/system/sysbox-fs.service
...
ExecStart=/usr/bin/sysbox-fs --mountpoint /mnt/mymount/var/lib/sysboxfs
...

~# docker info
Client: Docker Engine - Community
 Version:    28.3.0
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.25.0
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.37.3
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 1
 Server Version: 28.3.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc sysbox-runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05044ec0a9a75232cad458027ca83437aae3f4da
 runc version: v1.2.5-0-g59923ef
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.1.107
 Operating System: Ubuntu 22.04.1 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 13.32GiB
 Name: localhost.localdomain
 ID: f3a21194-c6a0-2224-b55d-29e9eee64f32
 Docker Root Dir: /mnt/mymount/var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false
 Default Address Pools:
   Base: 172.25.0.0/16, Size: 24

WARNING: No io.weight support
WARNING: No io.weight (per device) support
WARNING: No io.max (rbps) support
WARNING: No io.max (wbps) support
WARNING: No io.max (riops) support
WARNING: No io.max (wiops) support

[yogch]

Hi there, Can you please help?

[ctalledo]

Hi @yogch, apologies for the belated response.

docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: container_linux.go:439: starting container process caused: process_linux.go:608: container init caused: rootfs_linux.go:119: jailing process inside rootfs caused: permission denied: unknown

I suspect this error is caused by the Environment="DOCKER_RAMDISK=true" config you passed to Docker.

That env var causes Docker to tell Sysbox not to use pivot_root when jailing the container's init process, so instead Sysbox uses the traditional chroot (see code here).

There must be a bug in Sysbox that is preventing that traditional chroot from working correctly, and it has gone undetected because we haven't tested with DOCKER_RAMDISK=true.

Question: as a work-around, can you do away with DOCKER_RAMDISK=true in your setup?

[yogch]

@ctalledo thank you very much for your reply. I cannot remove DOCKER_RAMDISK=true. In case I try, when trying to run a container I get this:

# docker run --rm -it hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
17eec7bbc9d7: Pull complete
Digest: sha256:a0dfb02aac212703bfcb339d77d47ec32c8706ff250850ecc0e19c8737b18567
Status: Downloaded newer image for hello-world:latest
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error jailing process inside rootfs: pivot_root .: invalid argument: unknown

Run 'docker run --help' for more information

[yogch]

Hi @ctalledo Can you please let me know - are there any updates regarding this issue?

Thank you.