using loop device from wrapped app - can this work?

[x3nb63]

I am trying to wrap a app that needs to setup its own loop device, roughly it does:

1. create a file.img
2. mknod /dev/loopN ... (cause /dev is either a unmanaged tmpfs or created by bwrap --dev /dev ...)
3. losetup /dev/loopN file.img
4. mkexfatfs /dev/loopN ...
5. write data to /dev/loopN

this far I only got to step 3. whith bwrapping this: no matter what, losetup gets a Permission denied when opening the device file. (that I figured with strace). It also gets denied on /dev/loop-control if I happen to create this in step 2 (chmod a+rw was done, verified with ls).

I went as far as using the bubblewrap-suid alternate package in Arch Linux and runing bwrap itself as root plus providing a sudoers for the wrapped app, so it does sudo losetup ... which does not print any complain. Printing id and touching files from the wrapped app shows that its indeed root (0) and therefore should do.

Now I am at the point I don't understand anything anymore, since that app works in a privileged docker container (with its own tmpfs on /dev and CAP_MKNOD - thats all).

Can work what I want here?

[rusty-snake]

You:

that app works with CAP_MKNOD

Manpage:

By default no caps are left in the sandboxed process.

Did you tried with --cap-add ALL?

[x3nb63]

Did you tried with --cap-add ALL?

yes. And both, CAP_MKNOD and ALL, take me to step 3.

(there is a losetup -a between step 2 and 3 which fails without --cap-add)

here is what strace has to say when I create a /dev/loop-control in advance:

309709 execve("/opt/app/...", ["/opt/app.bin...", <many params>], 0x7ffc808bf0e8 /* 16 vars */) = 0
...
309713 execve("/sbin/losetup", ["/sbin/losetup", "-fP", "/home/user/storage/images"...], 0x7ffe582fff88 /* 16 vars */) = 0
309713 brk(NULL)                        = 0x562608643000
309713 arch_prctl(0x3001 /* ARCH_??? */, 0x7ffc45ec8980) = -1 EINVAL (Invalid argument)
309713 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
309713 openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 4
309713 newfstatat(4, "", {st_mode=S_IFREG|0644, st_size=165571, ...}, AT_EMPTY_PATH) = 0
309713 mmap(NULL, 165571, PROT_READ, MAP_PRIVATE, 4, 0) = 0x7fd2bb03f000
309713 close(4)                         = 0
309713 openat(AT_FDCWD, "/usr/lib/libsmartcols.so.1", O_RDONLY|O_CLOEXEC) = 4
309713 read(4, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
309713 newfstatat(4, "", {st_mode=S_IFREG|0755, st_size=104392, ...}, AT_EMPTY_PATH) = 0
309713 mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd2bb03d000
309713 mmap(NULL, 106520, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 4, 0) = 0x7fd2bb022000
309713 mmap(0x7fd2bb027000, 61440, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 4, 0x5000) = 0x7fd2bb027000
309713 mmap(0x7fd2bb036000, 20480, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 4, 0x14000) = 0x7fd2bb036000
309713 mmap(0x7fd2bb03b000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 4, 0x18000) = 0x7fd2bb03b000
309713 close(4)                         = 0
309713 openat(AT_FDCWD, "/usr/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = 4
309713 read(4, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P\224\2\0\0\0\0\0"..., 832) = 832
309713 pread64(4, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784
309713 pread64(4, "\4\0\0\0@\0\0\0\5\0\0\0GNU\0\2\0\0\300\4\0\0\0\3\0\0\0\0\0\0\0"..., 80, 848) = 80
309713 pread64(4, "\4\0\0\0\24\0\0\0\3\0\0\0GNU\0`\337\35\363\37\2\247\262=\250>\216\371#5\230"..., 68, 928) = 68
309713 newfstatat(4, "", {st_mode=S_IFREG|0755, st_size=2100888, ...}, AT_EMPTY_PATH) = 0
309713 pread64(4, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784
309713 mmap(NULL, 2146032, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 4, 0) = 0x7fd2bae16000
309713 mprotect(0x7fd2bae3e000, 1904640, PROT_NONE) = 0
309713 mmap(0x7fd2bae3e000, 1540096, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 4, 0x28000) = 0x7fd2bae3e000
309713 mmap(0x7fd2bafb6000, 360448, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 4, 0x1a0000) = 0x7fd2bafb6000
309713 mmap(0x7fd2bb00f000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 4, 0x1f8000) = 0x7fd2bb00f000
309713 mmap(0x7fd2bb015000, 52976, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fd2bb015000
309713 close(4)                         = 0
309713 mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd2bae13000
309713 arch_prctl(ARCH_SET_FS, 0x7fd2bae13740) = 0
309713 set_tid_address(0x7fd2bae13a10)  = 309713
309713 set_robust_list(0x7fd2bae13a20, 24) = 0
309713 rseq(0x7fd2bae140e0, 0x20, 0, 0x53053053) = 0
309713 mprotect(0x7fd2bb00f000, 16384, PROT_READ) = 0
309713 mprotect(0x7fd2bb03b000, 4096, PROT_READ) = 0
309713 mprotect(0x5626079bc000, 4096, PROT_READ) = 0
309713 mprotect(0x7fd2bb09d000, 8192, PROT_READ) = 0
309713 prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
309713 munmap(0x7fd2bb03f000, 165571)   = 0
309713 getrandom("\xa3\x9b\x4f\x4d\xea\x72\xa2\x74", 8, GRND_NONBLOCK) = 8
309713 brk(NULL)                        = 0x562608643000
309713 brk(0x562608664000)              = 0x562608664000
309713 openat(AT_FDCWD, "/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 4
309713 newfstatat(4, "", {st_mode=S_IFREG|0644, st_size=3407104, ...}, AT_EMPTY_PATH) = 0
309713 mmap(NULL, 3407104, PROT_READ, MAP_PRIVATE, 4, 0) = 0x7fd2baad3000
309713 close(4)                         = 0
309713 newfstatat(AT_FDCWD, "/sys/block", {st_mode=S_IFDIR|0755, st_size=0, ...}, 0) = 0
309713 uname({sysname="Linux", nodename="node01", ...}) = 0
309713 newfstatat(AT_FDCWD, "/dev/loop-control", {st_mode=S_IFBLK|0666, st_rdev=makedev(0xa, 0xed), ...}, 0) = 0
309713 openat(AT_FDCWD, "/dev/loop-control", O_RDWR|O_CLOEXEC) = -1 EACCES (Permission denied)
309713 newfstatat(AT_FDCWD, "/dev/loop", 0x7ffc45ec83a0, 0) = -1 ENOENT (No such file or directory)
309713 newfstatat(AT_FDCWD, "/dev/loop0", {st_mode=S_IFBLK|0666, st_rdev=makedev(0x7, 0), ...}, 0) = 0
309713 newfstatat(AT_FDCWD, "/dev/loop0", {st_mode=S_IFBLK|0666, st_rdev=makedev(0x7, 0), ...}, 0) = 0
309713 openat(AT_FDCWD, "/sys/dev/block/7:0", O_RDONLY|O_CLOEXEC) = 4
309713 openat(4, "loop/offset", O_RDONLY|O_CLOEXEC) = 5
309713 fcntl(5, F_GETFL)                = 0x8000 (flags O_RDONLY|O_LARGEFILE)
309713 newfstatat(5, "", {st_mode=S_IFREG|0444, st_size=4096, ...}, AT_EMPTY_PATH) = 0
309713 read(5, "0\n", 4096)             = 2
309713 close(5)                         = 0
309713 close(4)                         = 0
...

here it gives up later, I can make another trace where loop-control does not exist and gets denied directly on loop0.

(i simplify here, actually step 3 involves creating a partition which then requires to make a /dev/loop0p1 where data is then written to - it does not get that far)

[x3nb63]

here is a mock.sh script, as far as I got it to work:

(note that I don't suspect a bug in bwrap, just wanna know if its suitable to run this - cause for me its way easier than having to build+run a container for every change done to the app mocked here)

#!/bin/bash -e

sudo bwrap \
   --bind /tmp/mock-storage-dir /mock-storage-dir \
   --cap-add ALL \
   --dev /dev \
   --die-with-parent \
   --dir /var \
   --proc /proc \
   --ro-bind $PWD $PWD \
   --ro-bind /bin /bin \
   --ro-bind /etc /etc \
   --ro-bind /lib /lib \
   --ro-bind /sbin /sbin \
   --ro-bind /sys/block /sys/block \
   --ro-bind /sys/dev/block /sys/dev/block \
   --ro-bind /sys/devices /sys/devices \
   --ro-bind /usr /usr \
   --ro-bind-try /lib64 /lib64 \
   --share-net \
   --symlink ../tmp var/tmp \
   --tmpfs /tmp \
   --unshare-uts \
   /usr/bin/bash -x -e << BUILDER_MOCK
echo BUILDER MOCK start
img=/mock-storage-dir/mockimage.img

id

sudo mknod "/dev/loop0" b 7 0
sudo mknod "/dev/loop1" b 7 1
sudo mknod "/dev/loop2" b 7 2
sudo mknod "/dev/loop3" b 7 3
sudo mknod "/dev/loop4" b 7 4
sudo mknod "/dev/loop5" b 7 5
sudo mknod "/dev/loop6" b 7 6
sudo mknod "/dev/loop7" b 7 7
sudo mknod /dev/loop-control b 10 237

ls -la /dev/loop*

losetup -a

/sbin/losetup -fP \$img

/usr/sbin/parted -s /dev/loop0 \
   mklabel msdos \
   mkpart primary NTFS 1MiB "100%"

cat /proc/partitions

/bin/mknod /dev/loop0p1 b 259 0

# TODO: add more steps

echo BUILDER MOCK end
BUILDER_MOCK

the output is this

(on ArchLinux with kernel 5.15.55-2-lts and the bubblewrap-suid package installed)

+ echo BUILDER MOCK start
BUILDER MOCK start
+ img=/mock-storage-dir/mockimage.img
+ id
uid=0(root) gid=0(root) groups=0(root)
+ sudo mknod /dev/loop0 b 7 0
+ sudo mknod /dev/loop1 b 7 1
+ sudo mknod /dev/loop2 b 7 2
+ sudo mknod /dev/loop3 b 7 3
+ sudo mknod /dev/loop4 b 7 4
+ sudo mknod /dev/loop5 b 7 5
+ sudo mknod /dev/loop6 b 7 6
+ sudo mknod /dev/loop7 b 7 7
+ sudo mknod /dev/loop-control b 10 237
+ ls -la /dev/loop0 /dev/loop1 /dev/loop2 /dev/loop3 /dev/loop4 /dev/loop5 /dev/loop6 /dev/loop7 /dev/loop-control
brw-r--r-- 1 root root  7,   0 Aug  2 16:27 /dev/loop0
brw-r--r-- 1 root root  7,   1 Aug  2 16:27 /dev/loop1
brw-r--r-- 1 root root  7,   2 Aug  2 16:27 /dev/loop2
brw-r--r-- 1 root root  7,   3 Aug  2 16:27 /dev/loop3
brw-r--r-- 1 root root  7,   4 Aug  2 16:27 /dev/loop4
brw-r--r-- 1 root root  7,   5 Aug  2 16:27 /dev/loop5
brw-r--r-- 1 root root  7,   6 Aug  2 16:27 /dev/loop6
brw-r--r-- 1 root root  7,   7 Aug  2 16:27 /dev/loop7
brw-r--r-- 1 root root 10, 237 Aug  2 16:27 /dev/loop-control
+ losetup -a
+ /sbin/losetup -fP /mock-storage-dir/mockimage.img
losetup: /mock-storage-dir/mockimage.img: failed to set up loop device: Permission denied

[x3nb63]

crafted a app.mock binary from the example in loop(4) man page; it runs locally (as root) as well as in a privileged docker container.

however, I don't succeed in bwrap'in it. Always Permission denied on either /dev/loop-control or the loop device itself.

appears pretty much like #516 (did not try the "double wrapping" yet.)

I tend to conclude that there is no way around privileged for dealing with loop devices or block devices in general. Thus, bwrap is the wrong tool to constrain such an app, right?

[smcv]

I tend to conclude that there is no way around privileged for dealing with loop devices or block devices in general. Thus, bwrap is the wrong tool to constrain such an app, right?

Yes, I believe this is correct. If a resource isn't namespaced, then you need elevated capabilities in the "initial user namespace" (the user namespace that is created before the kernel hands over control to the global process 1, normally systemd or sysvinit) to be able to access it, and bwrap will not give you those.

I don't think loop(4) devices are namespaced, which means it would be a serious security vulnerability if unprivileged programs could manipulate them.

I'd recommend using a virtual machine, which will have its own set of loop devices that don't exist on the host system, for all your disk-image-building needs. You might find https://github.com/go-debos/fakemachine and https://github.com/go-debos/debos/ interesting.