Can't listen on privileged ports with `--dev /dev`

[jvns]

trafficstars

When I run bwrap with --dev /dev, I can't listen on a privileged port like 80. Here's an example, with an strace to show that the bind system call is failing:

$ strace -f -e bind bwrap --dev /dev --ro-bind / / --unshare-net --cap-add cap_net_bind_service nc -l -p 80
[pid 378942] bind(3, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("0.0.0.0")}, 16) = -1 EACCES (Permission denied)
Error: Couldn't setup listening socket (err=-3)

Without --dev /dev, it works fine:

$ strace -f -e bind bwrap --ro-bind / / --unshare-net --cap-add cap_net_bind_service nc -l -p 80
[pid 378903] bind(3, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("0.0.0.0")}, 16) = 0

In both cases the capabilities seem to be set up correctly, so I'm very puzzled as to how --dev /dev could possibly affect this.

This is with bubblewrap 0.6.1 and I'm running Linux 5.15.28. I also reproduced this issue on a machine running Linux 5.12.2.

[jvns]

I did a little more experimentation and discovered a workaround: if I nest two bwrap calls then I can listen on a privileged port in the new network namespace, even if I pass --dev /dev to the first one, like this:

 bwrap \
     --dev /dev \
     --proc /proc \
     --bind / / \
     bwrap  \
     --bind / / \
     --cap-add cap_net_bind_service \
     --unshare-net \
     nc -l -p 80

[rusty-snake]

bwrap
--dev /dev
--proc /proc
--bind / / \

If you place a mount over / behind a mount over /foo (--dev /dev --bind / /) the later "over-mounts" it i.e. --dev /dev is effectively a no-op.

edit: but in OP it's the same, make no sense.

[rusty-snake]

$ strace -f -e bind bwrap --dev /dev --ro-bind / / --unshare-net --cap-add cap_net_bind_service nc -l -p 80
strace: Process 305301 attached
[pid 305301] bind(4, {sa_family=AF_NETLINK, nl_pid=305301, nl_groups=00000000}, 12) = 0
[pid 305301] bind(3, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, 12) = 0
[pid 305301] bind(3, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, 12) = 0
Ncat: Failed to resolve default IPv4 address: Name or service not known. QUITTING.
[pid 305301] +++ exited with 2 +++
+++ exited with 2 +++

OS: Fedora Linux 36 bubblewrap: 0.5.0 ncat: 7.92 (nmap)

[jvns]

re the --dev /dev in the original example being a no-op: that's a good point, here's a more realistic reproduction of the same issue. This is on Manjaro, using GNU netcat 0.7.1 and bubblewrap 0.6.1

$ strace -f -e bind bwrap --ro-bind /usr /usr  --ro-bind /lib64 /lib64 --unshare-net --cap-add cap_net_bind_service nc -l -p 80
[pid 507797] bind(3, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("0.0.0.0")}, 16) = 0
$ strace -f -e bind bwrap --dev /dev --ro-bind /usr /usr  --ro-bind /lib64 /lib64 --unshare-net --cap-add cap_net_bind_service nc -l -p 80
[pid 507729] bind(3, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("0.0.0.0")}, 16) = -1 EACCES (Permission denied)
Error: Couldn't setup listening socket (err=-3)

[rusty-snake]

$ # nmap netcat:
$ strace -f -e bind bwrap --ro-bind /etc /etc --ro-bind /usr /usr --symlink usr/bin /bin --symlink usr/lib /lib --symlink usr/lib64 /lib64 --dev /dev --unshare-net --cap-add cap_net_bind_service ncat -l -p 80
strace: Process 318677 attached
[pid 318677] bind(4, {sa_family=AF_NETLINK, nl_pid=318677, nl_groups=00000000}, 12) = 0
[pid 318677] bind(3, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, 12) = 0
[pid 318677] bind(3, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, 12) = 0
Ncat: Failed to resolve default IPv4 address: Name or service not known. QUITTING.
[pid 318677] +++ exited with 2 +++
+++ exited with 2 +++
$ # OpenBSD netcat:
$ strace -f -e bind bwrap --ro-bind /etc /etc --ro-bind /usr /usr --symlink usr/bin /bin --symlink usr/lib /lib --symlink usr/lib64 /lib64 --dev /dev --unshare-net --cap-add cap_net_bind_service netcat -l -p 80
strace: Process 318685 attached
[pid 318685] bind(4, {sa_family=AF_NETLINK, nl_pid=318685, nl_groups=00000000}, 12) = 0
netcat: cannot use -p and -l
[pid 318685] +++ exited with 1 +++
+++ exited with 1 +++

Either it's because of GNU netcat (which seems to use AF_INET rather than AF_NETLINK as nmap/OpenBSD netcat do; and all seem to have an incompatible cli) or bubblewrap version (unlikely because of less changes) or distro/config.

[jvns]

That's weird, I get the exact same results with nmap ncat as with GNU netcat:

$ ncat -v
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: You must specify a host to connect to. QUITTING.
$ strace -f -e bind bwrap --dev /dev --ro-bind /usr /usr  --ro-bind /lib64 /lib64 --unshare-net --cap-add cap_net_bind_service ncat -l -p 80
[pid 508905] bind(4, {sa_family=AF_NETLINK, nl_pid=508905, nl_groups=00000000}, 12) = 0
[pid 508905] bind(3, {sa_family=AF_INET6, sin6_port=htons(80), sin6_flowinfo=htonl(0), inet_pton(AF_INET6, "::", &sin6_addr), sin6_scope_id=0}, 28) = -1 EACCES (Permission denied)
Ncat: bind to :::80: Permission denied. QUITTING.
$ strace -f -e bind bwrap  --ro-bind /usr /usr  --ro-bind /lib64 /lib64 --unshare-net --cap-add cap_net_bind_service ncat -l -p 80
[pid 508928] bind(4, {sa_family=AF_NETLINK, nl_pid=508928, nl_groups=00000000}, 12) = 0
[pid 508928] bind(3, {sa_family=AF_INET6, sin6_port=htons(80), sin6_flowinfo=htonl(0), inet_pton(AF_INET6, "::", &sin6_addr), sin6_scope_id=0}, 28) = 0
[pid 508928] bind(4, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("0.0.0.0")}, 16) = 0

I think the AF_NETLINK stuff is unrelated -- I left it out of my original output because AFAIK AF_NETLINK isn't what's used when opening a TCP connection, it's AF_INET or AF_INET6.

I think that your netcat would use AF_INET, but instead it's failing at an earlier stage (this "Failed to resolve default IPv4 address" error, or having different command line arguments).

[rusty-snake]

Yes, at least the first bind call seems to come from bwrap it self (or ld.so or libc or or at least it happens with every program).

[jvns]

I don't know if this is related or not, but I noticed that if --dev /dev is passed then the uid/gid mapping works a little differently (here in bubblewrap.c).

[mfila]

Same behavior on Linux Mint 20.3 with kernel 5.4.0 and both bwrap 0.4.0 (installed as a system package) and brwrap 0.6.2 (compiled from source):

$ lsb_release -d
Description:	Linux Mint 20.3
$
$ uname -r
5.4.0-121-generic
$
$ bwrap --version
bubblewrap 0.4.0
$
$ strace -f -e bind bwrap --dev /dev --ro-bind /usr /usr --symlink /usr/lib /lib --symlink /usr/lib64 /lib64 --unshare-net --cap-add cap_net_bind_service nc.openbsd -l -p 80
strace: Process 8787 attached
[pid  8787] bind(4, {sa_family=AF_NETLINK, nl_pid=8787, nl_groups=00000000}, 12) = 0
[pid  8787] bind(3, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("0.0.0.0")}, 16) = -1 EACCES (Permission denied)
nc.openbsd: Permission denied
[pid  8787] +++ exited with 1 +++
+++ exited with 1 +++
$
$ ./bwrap --version 
bubblewrap 0.6.2
$
$ strace -f -e bind ./bwrap --dev /dev --ro-bind /usr /usr --symlink /usr/lib /lib --symlink /usr/lib64 /lib64 --unshare-net --cap-add cap_net_bind_service nc.openbsd -l -p 80
strace: Process 8813 attached
[pid  8813] bind(4, {sa_family=AF_NETLINK, nl_pid=8813, nl_groups=00000000}, 12) = 0
[pid  8813] bind(3, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("0.0.0.0")}, 16) = -1 EACCES (Permission denied)
nc.openbsd: Permission denied
[pid  8813] +++ exited with 1 +++
+++ exited with 1 +++

[igo95862]

I don't know if this is related or not, but I noticed that if --dev /dev is passed then the uid/gid mapping works a little differently (here in bubblewrap.c).

It is because the bubblewrap will create an extra staging user namespace that is disconnected from any PID when --dev is passed.

You can easily see that with lsns --tree command:

├─4026532511          user        0                   
│ ├─4026532517     user        3       2620

You need to enter the disconnected namespace first. (with NS_GET_PARENT ioctl )