sysbox
sysbox copied to clipboard
Published
20 hours ago •
nestybox
nestybox
Trouble exposing devices in Sysbox containers with Docker's `--volume` option
My current setup with jenkins/docker is:
- Ubuntu host machine with docker installed
- Jenkins running on the host Ubuntu system as a docker container with
/var/run/docker.sockmounted into it via --volumes - Jenkins controller running builds inside Ubuntu-based docker containers with some additional tools installed on them, such as a python package called "usbhub", along with some runtime args:
args '--group-add=46 --device-cgroup-rule="c 189:* rmw" --volume /dev/bus/usb:/dev/bus/usb(189 being the usb major number) And with this setup, I'm able run things likeusbhub idand have it successfully identify the connected usb hub:
+ usbhub id
Hub Key : 624C (0)
MPN : CRZRYC
Revision : 2
Serial : 682719FFFEA6624C
USB Path : 1-123
Now when I try to instead run it as a sysbox container by replacing -v /var/run/docker.sock:/var/run/docker.sock with runtime=sysbox-runc and manually starting docker up inside the jenkins sysbox container with a docker exec call, I'm able to run docker commands properly the same way I was able to when mounting the socket directly into a normal docker container, but I get the following I/O error inside the build (Ubuntu) container when trying to communicate with my external usb device
*this is with /dev/bus/usb mounted into both the jenkins sysbox container from host as well as into the ubuntu build container
+ usbhub id
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/capablerobot_usbhub/i2c.py", line 164, in read_i2c_block_data
length = self.hub.handle.ctrl_transfer(REQ_OUT+1, self.CMD_I2C_WRITE, cmd, 0, [register], timeout=self.timeout)
File "/usr/local/lib/python3.8/dist-packages/usb/core.py", line 1071, in ctrl_transfer
self._ctx.managed_open()
File "/usr/local/lib/python3.8/dist-packages/usb/core.py", line 113, in wrapper
return f(self, *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/usb/core.py", line 131, in managed_open
self.handle = self.backend.open_device(self.dev)
File "/usr/local/lib/python3.8/dist-packages/usb/backend/libusb1.py", line 804, in open_device
return _DeviceHandle(dev)
File "/usr/local/lib/python3.8/dist-packages/usb/backend/libusb1.py", line 652, in __init__
_check(_lib.libusb_open(self.devid, byref(self.handle)))
File "/usr/local/lib/python3.8/dist-packages/usb/backend/libusb1.py", line 604, in _check
raise USBError(_strerror(ret), ret, _libusb_errno[ret])
usb.core.USBError: [Errno 5] Input/Output Error
@rodnymolina found a workaround using --device=/dev/bus/usb/001 instead of --volume /dev/bus/usb:/dev/bus/usb and running sudo chmod 666 /dev/bus/usb/001/001 and sudo chmod 666 /dev/bus/usb/001/002 for both the "nested" usb device directories in the host system before running the sysbox container. The problem with this is that the target devices need to be power cycled during multiple stages in the CI runs and come up in different modes as a "different device" at different times during the build, so being able to specify a parent directory under which they will all exist and reappear inside of is pretty necessary for my use case.
A couple of comments:
-
Normally the way you expose host devices inside a Docker container is via
--deviceflag, though I get why this is a problem in your setup (i.e., devices assigned different logical identifiers across reboots). However, there may be other mechanisms in Linux to prevent this. -
When exposing the device into the container via
--volume /dev/bus/usb:/dev/bus/usb, I am a bit surprised it does not work, given that Sysbox should have performed that mount and setup ID-shifting on it (via shiftfs or ID-mapped mounts), such that the device should show up with proper ownership (i.e.,root:root) inside the container.
What's the output of ls -l /dev/bus/usb inside the Sysbox container?
Some relevant output from within the sysbox container before switching to --device and chmod'ing the files listed above.
root@3f3b3bcc3f91:/# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
root@3f3b3bcc3f91:/# ls -l /dev
total 0
drwxr-xr-x 3 root root 60 Apr 14 19:13 bus
lrwxrwxrwx 1 root root 11 Apr 14 19:13 core -> /proc/kcore
lrwxrwxrwx 1 root root 13 Apr 14 19:13 fd -> /proc/self/fd
crw-rw-rw- 1 nobody nogroup 1, 7 Apr 4 22:52 full
crw-rw-rw- 1 root root 1, 3 Apr 4 22:52 kmsg
drwxrwxrwt 2 root nogroup 40 Apr 14 19:13 mqueue
crw-rw-rw- 1 root root 1, 3 Apr 4 22:52 null
lrwxrwxrwx 1 root root 8 Apr 14 19:13 ptmx -> pts/ptmx
drwxr-xr-x 2 root root 0 Apr 14 19:13 pts
crw-rw-rw- 1 nobody nogroup 1, 8 Apr 4 22:52 random
drwxrwxrwt 2 root root 40 Apr 14 19:13 shm
lrwxrwxrwx 1 root root 15 Apr 14 19:13 stderr -> /proc/self/fd/2
lrwxrwxrwx 1 root root 15 Apr 14 19:13 stdin -> /proc/self/fd/0
lrwxrwxrwx 1 root root 15 Apr 14 19:13 stdout -> /proc/self/fd/1
crw-rw-rw- 1 nobody nogroup 5, 0 Apr 13 18:57 tty
crw-rw-rw- 1 nobody nogroup 1, 9 Apr 4 22:52 urandom
crw-rw-rw- 1 nobody nogroup 1, 5 Apr 4 22:52 zero
root@3f3b3bcc3f91:/# ls -l /dev/bus/usb
total 0
drwxr-xr-x 2 nobody nogroup 280 Apr 13 21:34 001
drwxr-xr-x 2 nobody nogroup 60 Apr 6 19:34 002
root@3f3b3bcc3f91:/# findmnt
TARGET SOURCE FSTYPE OPTIONS
/ . shiftfs rw,relatime
├─/sys sysfs sysfs rw,nosuid,nodev,noexec,relatime
│ ├─/sys/firmware tmpfs tmpfs ro,relatime,uid=165536,gid=165536,inode6
│ ├─/sys/fs/cgroup tmpfs tmpfs rw,nosuid,nodev,noexec,relatime,mode=755
│ │ ├─/sys/fs/cgroup/systemd systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,na
│ │ ├─/sys/fs/cgroup/devices cgroup cgroup rw,nosuid,nodev,noexec,relatime,devices
│ │ ├─/sys/fs/cgroup/perf_event cgroup cgroup rw,nosuid,nodev,noexec,relatime,perf_eve
│ │ ├─/sys/fs/cgroup/pids cgroup cgroup rw,nosuid,nodev,noexec,relatime,pids
│ │ ├─/sys/fs/cgroup/hugetlb cgroup cgroup rw,nosuid,nodev,noexec,relatime,hugetlb
│ │ ├─/sys/fs/cgroup/cpu,cpuacct cgroup cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpua
│ │ ├─/sys/fs/cgroup/blkio cgroup cgroup rw,nosuid,nodev,noexec,relatime,blkio
│ │ ├─/sys/fs/cgroup/freezer cgroup cgroup rw,nosuid,nodev,noexec,relatime,freezer
│ │ ├─/sys/fs/cgroup/misc cgroup cgroup rw,nosuid,nodev,noexec,relatime,misc
│ │ ├─/sys/fs/cgroup/rdma cgroup cgroup rw,nosuid,nodev,noexec,relatime,rdma
│ │ ├─/sys/fs/cgroup/memory cgroup cgroup rw,nosuid,nodev,noexec,relatime,memory
│ │ ├─/sys/fs/cgroup/net_cls,net_prio cgroup cgroup rw,nosuid,nodev,noexec,relatime,net_cls,
│ │ └─/sys/fs/cgroup/cpuset cgroup cgroup rw,nosuid,nodev,noexec,relatime,cpuset
│ ├─/sys/kernel/config tmpfs tmpfs rw,nosuid,nodev,noexec,relatime,size=102
│ ├─/sys/kernel/debug tmpfs tmpfs rw,nosuid,nodev,noexec,relatime,size=102
│ ├─/sys/kernel/tracing tmpfs tmpfs rw,nosuid,nodev,noexec,relatime,size=102
│ ├─/sys/devices/virtual/dmi/id/product_uuid
│ │ sysboxfs[/sys/devices/virtual/dmi/id/product_uuid] fuse rw,nosuid,nodev,relatime,user_id=0,group
│ └─/sys/module/nf_conntrack/parameters/hashsize
│ sysboxfs[/sys/module/nf_conntrack/parameters/hashsize] fuse rw,nosuid,nodev,relatime,user_id=0,group
├─/proc proc proc rw,nosuid,nodev,noexec,relatime
│ ├─/proc/bus proc[/bus] proc ro,nosuid,nodev,noexec,relatime
│ ├─/proc/fs proc[/fs] proc ro,nosuid,nodev,noexec,relatime
│ ├─/proc/irq proc[/irq] proc ro,nosuid,nodev,noexec,relatime
│ ├─/proc/sysrq-trigger proc[/sysrq-trigger] proc ro,nosuid,nodev,noexec,relatime
│ ├─/proc/asound tmpfs tmpfs ro,relatime,uid=165536,gid=165536,inode6
│ ├─/proc/acpi tmpfs tmpfs ro,relatime,uid=165536,gid=165536,inode6
│ ├─/proc/keys udev[/null] devtmpfs rw,nosuid,noexec,relatime,size=2938712k,
│ ├─/proc/timer_list udev[/null] devtmpfs rw,nosuid,noexec,relatime,size=2938712k,
│ ├─/proc/scsi tmpfs tmpfs ro,relatime,uid=165536,gid=165536,inode6
│ ├─/proc/swaps sysboxfs[/proc/swaps] fuse rw,nosuid,nodev,relatime,user_id=0,group
│ ├─/proc/sys sysboxfs[/proc/sys] fuse rw,nosuid,nodev,relatime,user_id=0,group
│ └─/proc/uptime sysboxfs[/proc/uptime] fuse rw,nosuid,nodev,relatime,user_id=0,group
├─/dev tmpfs tmpfs rw,nosuid,size=65536k,mode=755,uid=16553
│ ├─/dev/mqueue mqueue mqueue rw,nosuid,nodev,noexec,relatime
│ ├─/dev/pts devpts devpts rw,nosuid,noexec,relatime,gid=165541,mod
│ ├─/dev/shm shm tmpfs rw,nosuid,nodev,noexec,relatime,size=655
│ ├─/dev/null udev[/null] devtmpfs rw,nosuid,noexec,relatime,size=2938712k,
│ ├─/dev/bus/usb udev[/bus/usb] devtmpfs rw,nosuid,noexec,relatime,size=2938712k,
│ ├─/dev/kmsg udev[/null] devtmpfs rw,nosuid,noexec,relatime,size=2938712k,
│ ├─/dev/random udev[/random] devtmpfs rw,nosuid,noexec,relatime,size=2938712k,
│ ├─/dev/full udev[/full] devtmpfs rw,nosuid,noexec,relatime,size=2938712k,
│ ├─/dev/tty udev[/tty] devtmpfs rw,nosuid,noexec,relatime,size=2938712k,
│ ├─/dev/zero udev[/zero] devtmpfs rw,nosuid,noexec,relatime,size=2938712k,
│ └─/dev/urandom udev[/urandom] devtmpfs rw,nosuid,noexec,relatime,size=2938712k,
├─/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs
│ /dev/sda2[/var/lib/sysbox/containerd/3f3b3bcc3f9144c23b244fb4417c7a638d226b167add9f4948edc9da019772bc]
│ ext4 rw,relatime,errors=remount-ro
├─/var/jenkins_home /dev/sda2[/var/lib/docker/volumes/gsg-jenkins_jenkins-data/_data]
│ ext4 rw,relatime,idmapped,errors=remount-ro
├─/etc/resolv.conf /dev/sda2[/var/lib/docker/containers/3f3b3bcc3f9144c23b244fb4417c7a638d226b167add9f4948edc9da019772bc/resolv.conf]
│ ext4 rw,relatime,idmapped,errors=remount-ro
├─/etc/hostname /dev/sda2[/var/lib/docker/containers/3f3b3bcc3f9144c23b244fb4417c7a638d226b167add9f4948edc9da019772bc/hostname]
│ ext4 rw,relatime,idmapped,errors=remount-ro
├─/etc/hosts /dev/sda2[/var/lib/docker/containers/3f3b3bcc3f9144c23b244fb4417c7a638d226b167add9f4948edc9da019772bc/hosts]
│ ext4 rw,relatime,idmapped,errors=remount-ro
├─/var/lib/docker /dev/sda2[/var/lib/sysbox/docker/3f3b3bcc3f9144c23b244fb4417c7a638d226b167add9f4948edc9da019772bc]
│ ext4 rw,relatime,errors=remount-ro
├─/var/lib/rancher/k3s /dev/sda2[/var/lib/sysbox/rancher-k3s/3f3b3bcc3f9144c23b244fb4417c7a638d226b167add9f4948edc9da019772bc]
│ ext4 rw,relatime,errors=remount-ro
├─/var/lib/rancher/rke2 /dev/sda2[/var/lib/sysbox/rancher-rke2/3f3b3bcc3f9144c23b244fb4417c7a638d226b167add9f4948edc9da019772bc]
│ ext4 rw,relatime,errors=remount-ro
├─/var/lib/kubelet /dev/sda2[/var/lib/sysbox/kubelet/3f3b3bcc3f9144c23b244fb4417c7a638d226b167add9f4948edc9da019772bc]
│ ext4 rw,relatime,errors=remount-ro
├─/var/lib/k0s /dev/sda2[/var/lib/sysbox/k0s/3f3b3bcc3f9144c23b244fb4417c7a638d226b167add9f4948edc9da019772bc]
│ ext4 rw,relatime,errors=remount-ro
├─/var/lib/buildkit /dev/sda2[/var/lib/sysbox/buildkit/3f3b3bcc3f9144c23b244fb4417c7a638d226b167add9f4948edc9da019772bc]
│ ext4 rw,relatime,errors=remount-ro
├─/usr/src/linux-headers-5.13.0-37-generic
│ /dev/sda2[/usr/src/linux-headers-5.13.0-37-generic] ext4 ro,relatime,idmapped,errors=remount-ro
├─/usr/src/linux-hwe-5.13-headers-5.13.0-37
│ /dev/sda2[/usr/src/linux-hwe-5.13-headers-5.13.0-37] ext4 ro,relatime,idmapped,errors=remount-ro
└─/lib/modules/5.13.0-37-generic /dev/sda2[/usr/lib/modules/5.13.0-37-generic] ext4 ro,relatime,idmapped,errors=remount-ro
Some relevant output from within the sysbox container before switching to --device and chmod'ing the files listed above.
What was the command you used to create this container (e.g., docker run --runtime=sysbox-runc ...) ?
I'm using docker-compose build --build-arg DOCKER_GROUP_ID=xxx to build the image then running docker-compose up --no-build --detached along with the following docker-compose.yml to run the container:
version: '3'
services:
jenkins:
build:
context: .
dockerfile: Dockerfile
args:
- DOCKER_GROUP_ID
container_name: jenkins-controller
environment:
- JENKINS_ADMIN_ID
- JENKINS_ADMIN_PASSWORD
- GH_APP_KEY
runtime: sysbox-runc
ports:
- '8080:8080'
- '50000:50000'
volumes:
- jenkins-data:/var/jenkins_home
- /dev/bus/usb:/dev/bus/usb
volumes:
jenkins-data:
Thanks @grvvy. Could you paste this line in full (it seems cut-off):
├─/dev/bus/usb udev[/bus/usb] devtmpfs rw,nosuid,noexec,relatime,size=2938712k,
Sorry about that, here's the full findmnt output
gsgadmin@testserver:~$ stat -c %a /dev/bus/usb/001
755
gsgadmin@testserver:~$ stat -c %a /dev/bus/usb/001/001
666
gsgadmin@testserver:~$ stat -c %a /dev/bus/usb/001/002
666
gsgadmin@testserver:~$ docker exec -it -u root jenkins-controller /bin/bash
root@f90f2b64de60:/# stat -c %a /dev/bus/usb/001/001
666
root@f90f2b64de60:/# stat -c %a /dev/bus/usb/001/002
666
root@f90f2b64de60:/# stat -c %a /dev/bus/usb/001
755
root@f90f2b64de60:/# findmnt
TARGET SOURCE FSTYPE OPTIONS
/ . shiftfs rw,relatime
├─/sys sysfs sysfs rw,nosuid,nodev,noexec,relatime
│ ├─/sys/firmware tmpfs tmpfs ro,relatime,uid=165536,gid=165536,inode64
│ ├─/sys/fs/cgroup tmpfs tmpfs rw,nosuid,nodev,noexec,relatime,mode=755,uid=165536,gid=165536,inode64
│ │ ├─/sys/fs/cgroup/systemd systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd
│ │ ├─/sys/fs/cgroup/devices cgroup cgroup rw,nosuid,nodev,noexec,relatime,devices
│ │ ├─/sys/fs/cgroup/perf_event cgroup cgroup rw,nosuid,nodev,noexec,relatime,perf_event
│ │ ├─/sys/fs/cgroup/pids cgroup cgroup rw,nosuid,nodev,noexec,relatime,pids
│ │ ├─/sys/fs/cgroup/hugetlb cgroup cgroup rw,nosuid,nodev,noexec,relatime,hugetlb
│ │ ├─/sys/fs/cgroup/cpu,cpuacct cgroup cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct
│ │ ├─/sys/fs/cgroup/blkio cgroup cgroup rw,nosuid,nodev,noexec,relatime,blkio
│ │ ├─/sys/fs/cgroup/freezer cgroup cgroup rw,nosuid,nodev,noexec,relatime,freezer
│ │ ├─/sys/fs/cgroup/misc cgroup cgroup rw,nosuid,nodev,noexec,relatime,misc
│ │ ├─/sys/fs/cgroup/rdma cgroup cgroup rw,nosuid,nodev,noexec,relatime,rdma
│ │ ├─/sys/fs/cgroup/memory cgroup cgroup rw,nosuid,nodev,noexec,relatime,memory
│ │ ├─/sys/fs/cgroup/net_cls,net_prio cgroup cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio
│ │ └─/sys/fs/cgroup/cpuset cgroup cgroup rw,nosuid,nodev,noexec,relatime,cpuset
│ ├─/sys/kernel/config tmpfs tmpfs rw,nosuid,nodev,noexec,relatime,size=1024k,uid=165536,gid=165536,inode64
│ ├─/sys/kernel/debug tmpfs tmpfs rw,nosuid,nodev,noexec,relatime,size=1024k,uid=165536,gid=165536,inode64
│ ├─/sys/kernel/tracing tmpfs tmpfs rw,nosuid,nodev,noexec,relatime,size=1024k,uid=165536,gid=165536,inode64
│ ├─/sys/devices/virtual/dmi/id/product_uuid sysboxfs[/sys/devices/virtual/dmi/id/product_uuid] fuse rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other
│ └─/sys/module/nf_conntrack/parameters/hashsize sysboxfs[/sys/module/nf_conntrack/parameters/hashsize] fuse rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other
├─/proc proc proc rw,nosuid,nodev,noexec,relatime
│ ├─/proc/bus proc[/bus] proc ro,nosuid,nodev,noexec,relatime
│ ├─/proc/fs proc[/fs] proc ro,nosuid,nodev,noexec,relatime
│ ├─/proc/irq proc[/irq] proc ro,nosuid,nodev,noexec,relatime
│ ├─/proc/sysrq-trigger proc[/sysrq-trigger] proc ro,nosuid,nodev,noexec,relatime
│ ├─/proc/asound tmpfs tmpfs ro,relatime,uid=165536,gid=165536,inode64
│ ├─/proc/acpi tmpfs tmpfs ro,relatime,uid=165536,gid=165536,inode64
│ ├─/proc/keys udev[/null] devtmpfs rw,nosuid,noexec,relatime,size=2938712k,nr_inodes=734678,mode=755,inode64
│ ├─/proc/timer_list udev[/null] devtmpfs rw,nosuid,noexec,relatime,size=2938712k,nr_inodes=734678,mode=755,inode64
│ ├─/proc/scsi tmpfs tmpfs ro,relatime,uid=165536,gid=165536,inode64
│ ├─/proc/swaps sysboxfs[/proc/swaps] fuse rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other
│ ├─/proc/sys sysboxfs[/proc/sys] fuse rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other
│ └─/proc/uptime sysboxfs[/proc/uptime] fuse rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other
├─/dev tmpfs tmpfs rw,nosuid,size=65536k,mode=755,uid=165536,gid=165536,inode64
│ ├─/dev/mqueue mqueue mqueue rw,nosuid,nodev,noexec,relatime
│ ├─/dev/pts devpts devpts rw,nosuid,noexec,relatime,gid=165541,mode=620,ptmxmode=666
│ ├─/dev/shm shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k,uid=165536,gid=165536,inode64
│ ├─/dev/null udev[/null] devtmpfs rw,nosuid,noexec,relatime,size=2938712k,nr_inodes=734678,mode=755,inode64
│ ├─/dev/bus/usb udev[/bus/usb] devtmpfs rw,nosuid,noexec,relatime,size=2938712k,nr_inodes=734678,mode=755,inode64
│ ├─/dev/kmsg udev[/null] devtmpfs rw,nosuid,noexec,relatime,size=2938712k,nr_inodes=734678,mode=755,inode64
│ ├─/dev/random udev[/random] devtmpfs rw,nosuid,noexec,relatime,size=2938712k,nr_inodes=734678,mode=755,inode64
│ ├─/dev/full udev[/full] devtmpfs rw,nosuid,noexec,relatime,size=2938712k,nr_inodes=734678,mode=755,inode64
│ ├─/dev/tty udev[/tty] devtmpfs rw,nosuid,noexec,relatime,size=2938712k,nr_inodes=734678,mode=755,inode64
│ ├─/dev/zero udev[/zero] devtmpfs rw,nosuid,noexec,relatime,size=2938712k,nr_inodes=734678,mode=755,inode64
│ └─/dev/urandom udev[/urandom] devtmpfs rw,nosuid,noexec,relatime,size=2938712k,nr_inodes=734678,mode=755,inode64
├─/var/lib/rancher/k3s /dev/sda2[/var/lib/sysbox/rancher-k3s/f90f2b64de6035284853b241ff80a4d10ee64ddfc4381e8982e0f064661ee678]
│ ext4 rw,relatime,errors=remount-ro
├─/var/jenkins_home /dev/sda2[/var/lib/docker/volumes/gsg-jenkins_jenkins-data/_data] ext4 rw,relatime,idmapped,errors=remount-ro
├─/etc/resolv.conf /dev/sda2[/var/lib/docker/containers/f90f2b64de6035284853b241ff80a4d10ee64ddfc4381e8982e0f064661ee678/resolv.conf]
│ ext4 rw,relatime,idmapped,errors=remount-ro
├─/etc/hostname /dev/sda2[/var/lib/docker/containers/f90f2b64de6035284853b241ff80a4d10ee64ddfc4381e8982e0f064661ee678/hostname]
│ ext4 rw,relatime,idmapped,errors=remount-ro
├─/etc/hosts /dev/sda2[/var/lib/docker/containers/f90f2b64de6035284853b241ff80a4d10ee64ddfc4381e8982e0f064661ee678/hosts]
│ ext4 rw,relatime,idmapped,errors=remount-ro
├─/var/lib/rancher/rke2 /dev/sda2[/var/lib/sysbox/rancher-rke2/f90f2b64de6035284853b241ff80a4d10ee64ddfc4381e8982e0f064661ee678]
│ ext4 rw,relatime,errors=remount-ro
├─/var/lib/kubelet /dev/sda2[/var/lib/sysbox/kubelet/f90f2b64de6035284853b241ff80a4d10ee64ddfc4381e8982e0f064661ee678]
│ ext4 rw,relatime,errors=remount-ro
├─/var/lib/k0s /dev/sda2[/var/lib/sysbox/k0s/f90f2b64de6035284853b241ff80a4d10ee64ddfc4381e8982e0f064661ee678]
│ ext4 rw,relatime,errors=remount-ro
├─/var/lib/buildkit /dev/sda2[/var/lib/sysbox/buildkit/f90f2b64de6035284853b241ff80a4d10ee64ddfc4381e8982e0f064661ee678]
│ ext4 rw,relatime,errors=remount-ro
├─/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs /dev/sda2[/var/lib/sysbox/containerd/f90f2b64de6035284853b241ff80a4d10ee64ddfc4381e8982e0f064661ee678]
│ ext4 rw,relatime,errors=remount-ro
├─/var/lib/docker /dev/sda2[/var/lib/sysbox/docker/f90f2b64de6035284853b241ff80a4d10ee64ddfc4381e8982e0f064661ee678]
│ ext4 rw,relatime,errors=remount-ro
├─/usr/src/linux-headers-5.13.0-37-generic /dev/sda2[/usr/src/linux-headers-5.13.0-37-generic] ext4 ro,relatime,idmapped,errors=remount-ro
├─/usr/src/linux-hwe-5.13-headers-5.13.0-37 /dev/sda2[/usr/src/linux-hwe-5.13-headers-5.13.0-37] ext4 ro,relatime,idmapped,errors=remount-ro
└─/lib/modules/5.13.0-37-generic /dev/sda2[/usr/lib/modules/5.13.0-37-generic] ext4 ro,relatime,idmapped,errors=remount-ro