sysbox
sysbox copied to clipboard
nestybox
/proc/<PID>/maps paths are incorrect inside a Sysbox container with shiftfs (breaks apps like dotnet inside the container)
Hi,
I'm using sysbox for build containers to be able to generate Docker images inside a Docker container. Creating container images works, but for some reason the 'dotnet' command used to build C# .NET stuff doesn't.
It fails with the following error message indicating that it is using the wrong path to create a subprocess:
# dotnet publish -warnaserror --configuration Release -o build/server Server
Welcome to .NET 6.0!
---------------------
SDK Version: 6.0.400
Telemetry
---------
The .NET tools collect usage data in order to help us improve your experience. It is collected by Microsoft and shared with the community. You can opt-out of telemetry by setting the DOTNET_CLI_TELEMETRY_OPTOUT environment variable to '1' or 'true' using your favorite shell.
Read more about .NET CLI Tools telemetry: https://aka.ms/dotnet-cli-telemetry
----------------
Installed an ASP.NET Core HTTPS development certificate.
To trust the certificate run 'dotnet dev-certs https --trust' (Windows and macOS only).
Learn about HTTPS: https://aka.ms/dotnet-https
----------------
Write your first app: https://aka.ms/dotnet-hello-world
Find out what's new: https://aka.ms/dotnet-whats-new
Explore documentation: https://aka.ms/dotnet-docs
Report issues and find source on GitHub: https://github.com/dotnet/core
Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli
--------------------------------------------------------------------------------------
MSBuild version 17.3.0+92e077650 for .NET
Determining projects to restore...
Restored /home/jenkins/workspace/<Tool Name>/Server/Server.csproj (in 1.35 sec).
Restored /home/jenkins/workspace/<Tool Name>/Shared/Shared.csproj (in 2.74 sec).
/usr/share/dotnet/sdk/6.0.400/Roslyn/Microsoft.CSharp.Core.targets(75,5): error MSB6004: The specified task executable location "/var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/share/dotnet/dotnet" is invalid. [/home/jenkins/workspace/<Tool Name>/Shared/Shared.csproj]
Enabling more verbose output for dotnet shows that it appears to believe its path is /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/share/dotnet/dotnet instead of just /usr/share/dotnet/dotnet:
MSBuild version 17.3.0+92e077650 for .NET
/usr/share/dotnet/sdk/6.0.400/MSBuild.dll -distributedlogger:Microsoft.DotNet.Tools.MSBuild.MSBuildLogger,/usr/share/dotnet/sdk/6.0.400/dotnet.dll*Microsoft.DotNet.Tools.MSBuild.MSBuildForwardingLogger,/usr/share/dotnet/sdk/6.0.400/dotnet.dll -maxcpucount -property:PublishDir=/home/jenkins/workspace/<Tool Name>/build/server -property:Configuration=Release -restore -target:Publish -verbosity:m -verbosity:diag -warnaserror Server/Server.csproj
Build started 02/15/2023 08:35:15.
Environment at start of build:
HOSTNAME = be0b24a50b5e
JAVA_HOME = /opt/java/openjdk
LANG = C.UTF-8
DOTNET_HOST_PATH = /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/share/dotnet/dotnet
MSBuildExtensionsPath = /usr/share/dotnet/sdk/6.0.400/
SHLVL = 1
MSBuildSDKsPath = /usr/share/dotnet/sdk/6.0.400/Sdks
HOME = /root
PATH = /opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD = /home/jenkins/workspace/<Tool Name>
MSBuildLoadMicrosoftTargetsReadOnly = true
NODE_VERSION = 16.15.1
OLDPWD = /home/jenkins/workspace
TERM = xterm
_ = /usr/bin/dotnet
JENKINS_AGENT_HOME = /home/jenkins
08:35:15.033 0>Process = "/var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/share/dotnet/dotnet"
MSBuild executable path = "/usr/share/dotnet/sdk/6.0.400/MSBuild.dll"
Command line arguments = "/usr/share/dotnet/sdk/6.0.400/MSBuild.dll -maxcpucount -verbosity:m -restore -target:Publish -property:PublishDir=/home/jenkins/workspace/<Tool Name>/build/server -property:Configuration=Release -verbosity:diag -warnaserror Server -distributedlogger:Microsoft.DotNet.Tools.MSBuild.MSBuildLogger,/usr/share/dotnet/sdk/6.0.400/dotnet.dll*Microsoft.DotNet.Tools.MSBuild.MSBuildForwardingLogger,/usr/share/dotnet/sdk/6.0.400/dotnet.dll"
I've also tried manually setting the environment variable DOTNET_HOST_PATH to the correct value, but it still got overriden to the overlay2 one. When I run the same command using the same build container image without sysbox, the 'dotnet' command works as expected.
The host is a fully-patched Ubuntu 20.04 server using the following kernel: Linux be0b24a50b5e 5.4.0-139-generic #156-Ubuntu SMP Fri Jan 20 17:27:18 UTC 2023 x86_64 GNU/Linux
Sysbox 0.5.2-0.linux is used.
This is the output of the docker info command:
$ sudo docker info
Client:
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.10.2
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.16.0
Path: /usr/libexec/docker/cli-plugins/docker-compose
scan: Docker Scan (Docker Inc.)
Version: v0.23.0
Path: /usr/libexec/docker/cli-plugins/docker-scan
Server:
Containers: 12
Running: 1
Paused: 0
Stopped: 11
Images: 27
Server Version: 23.0.1
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc sysbox-runc io.containerd.runc.v2
Default Runtime: sysbox-runc
Init Binary: docker-init
containerd version: 31aa4358a36870b21a992d3ad2bef29e1d693bec
runc version:
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
Kernel Version: 5.4.0-139-generic
Operating System: Ubuntu 20.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 3.782GiB
Name: rtb-linuxbuild
ID: MWHW:2JPI:B4UT:6GQQ:L6Z4:B3IQ:5K66:2ZZ4:VQV5:OO5G:UMYI:QKTZ
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Default Address Pools:
Base: 172.25.0.0/16, Size: 24
WARNING: No swap limit support
Does anyone have any idea what exactly is going on here? It feels like somehow an OverlayFS path is getting exposed to a process in a way it probably shouldn't be.
I've done a bit more digging using strace and apparently the 'dotnet' command gets this path from reading /proc/PID/maps. I've looked at it in the container and it looks like this for the 'cat' command itself:
# cat /proc/self/maps
55a28617e000-55a286180000 r--p 00000000 00:56 1060266 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/bin/cat
55a286180000-55a286185000 r-xp 00002000 00:56 1060266 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/bin/cat
55a286185000-55a286188000 r--p 00007000 00:56 1060266 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/bin/cat
55a286188000-55a286189000 r--p 00009000 00:56 1060266 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/bin/cat
55a286189000-55a28618a000 rw-p 0000a000 00:56 1060266 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/bin/cat
55a286ca3000-55a286cc4000 rw-p 00000000 00:00 0 [heap]
7f551a0da000-7f551a0fc000 rw-p 00000000 00:00 0
7f551a0fc000-7f551a151000 r--p 00000000 00:56 1061864 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/lib/locale/C.UTF-8/LC_CTYPE
7f551a151000-7f551a152000 r--p 00000000 00:56 1061871 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/lib/locale/C.UTF-8/LC_NUMERIC
7f551a152000-7f551a153000 r--p 00000000 00:56 1061874 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/lib/locale/C.UTF-8/LC_TIME
7f551a153000-7f551a2c6000 r--p 00000000 00:56 1061863 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/lib/locale/C.UTF-8/LC_COLLATE
7f551a2c6000-7f551a2c7000 r--p 00000000 00:56 1061869 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/lib/locale/C.UTF-8/LC_MONETARY
7f551a2c7000-7f551a2c8000 r--p 00000000 00:56 1061868 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/lib/locale/C.UTF-8/LC_MESSAGES/SYS_LC_MESSAGES
7f551a2c8000-7f551a2cf000 r--s 00000000 00:56 1458798 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
7f551a2cf000-7f551a2f1000 r--p 00000000 00:56 1449723 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/lib/x86_64-linux-gnu/libc-2.31.so
7f551a2f1000-7f551a44b000 r-xp 00022000 00:56 1449723 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/lib/x86_64-linux-gnu/libc-2.31.so
7f551a44b000-7f551a49a000 r--p 0017c000 00:56 1449723 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/lib/x86_64-linux-gnu/libc-2.31.so
7f551a49a000-7f551a49e000 r--p 001ca000 00:56 1449723 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/lib/x86_64-linux-gnu/libc-2.31.so
7f551a49e000-7f551a4a0000 rw-p 001ce000 00:56 1449723 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/lib/x86_64-linux-gnu/libc-2.31.so
7f551a4a0000-7f551a4a6000 rw-p 00000000 00:00 0
7f551a4a6000-7f551a4a7000 r--p 00000000 00:56 1061872 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/lib/locale/C.UTF-8/LC_PAPER
7f551a4a7000-7f551a4a8000 r--p 00000000 00:56 1061870 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/lib/locale/C.UTF-8/LC_NAME
7f551a4a8000-7f551a4a9000 r--p 00000000 00:56 1061862 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/lib/locale/C.UTF-8/LC_ADDRESS
7f551a4a9000-7f551a4aa000 r--p 00000000 00:56 1061873 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/lib/locale/C.UTF-8/LC_TELEPHONE
7f551a4aa000-7f551a4ab000 r--p 00000000 00:56 1061866 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/lib/locale/C.UTF-8/LC_MEASUREMENT
7f551a4ab000-7f551a4ac000 r--p 00000000 00:56 1449510 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/lib/x86_64-linux-gnu/ld-2.31.so
7f551a4ac000-7f551a4cc000 r-xp 00001000 00:56 1449510 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/lib/x86_64-linux-gnu/ld-2.31.so
7f551a4cc000-7f551a4d4000 r--p 00021000 00:56 1449510 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/lib/x86_64-linux-gnu/ld-2.31.so
7f551a4d4000-7f551a4d5000 r--p 00000000 00:56 1061865 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/lib/locale/C.UTF-8/LC_IDENTIFICATION
7f551a4d5000-7f551a4d6000 r--p 00029000 00:56 1449510 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/lib/x86_64-linux-gnu/ld-2.31.so
7f551a4d6000-7f551a4d7000 rw-p 0002a000 00:56 1449510 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/lib/x86_64-linux-gnu/ld-2.31.so
7f551a4d7000-7f551a4d8000 rw-p 00000000 00:00 0
7ffd54194000-7ffd541b5000 rw-p 00000000 00:00 0 [stack]
7ffd541d8000-7ffd541db000 r--p 00000000 00:00 0 [vvar]
7ffd541db000-7ffd541dc000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0 [vsyscall]
I believe the OverlayFS paths in this file are paths from the Docker host that should probably not be exposed to processes running inside the sysbox container.
Hi @st0rmi, thanks for trying Sysbox and for filing the issue.
dotnet publish -warnaserror --configuration Release -o build/server Server
This command is running inside the sysbox container correct? Sounds like it is, just want to double check.
Enabling more verbose output for dotnet shows that it appears to believe its path is /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/usr/share/dotnet/dotnet instead of just /usr/share/dotnet/dotnet:
Got it.
I've done a bit more digging using strace and apparently the 'dotnet' command gets this path from reading /proc/PID/maps
I see.
I've looked at it in the container and it looks like this for the 'cat' command itself:
# cat /proc/self/maps
55a28617e000-55a286180000 r--p 00000000 00:56 1060266 /var/lib/docker/overlay2/8b868465148d0c96570d80f0fac60dfc5bc0ea7b2c82e27100933c46b0ece483/merged/bin/cat
That's strange; the paths under /proc/self/maps should be relative to the container's chroot jail, so it should have reported /bin/cat instead.
Here's what I see on my Ubuntu dev machine for example, when launching a container with Sysbox:
$ docker run --runtime=sysbox-runc -it --rm ubuntu
root@21a8be6a8b6e:/# cat /proc/self/maps
55e5afaf9000-55e5afafb000 r--p 00000000 00:a49 28363251 /usr/bin/cat
I believe the OverlayFS paths in this file are paths from the Docker host that should probably not be exposed to processes running inside the sysbox container.
Correct, the paths should be relative to the container's chroot jail. How can I reproduce on my side?
Hi @st0rmi, I was able to reproduce this issue by configuring the host machine with the shiftfs kernel module:
$ docker run --runtime=sysbox-runc -it --rm alpine
/ # cat /proc/self/maps
563ed2ed9000-563ed2ee5000 r--p 00000000 00:2c 1806417 /var/lib/docker/overlay2/6c6bd4bbe4cb9f0886d24c028d417d9ace5f11afc10964addfbb9ed50947ffde/merged/bin/busybox
563ed2ee5000-563ed2f81000 r-xp 0000c000 00:2c 1806417 /var/lib/docker/overlay2/6c6bd4bbe4cb9f0886d24c028d417d9ace5f11afc10964addfbb9ed50947ffde/merged/bin/busybox
563ed2f81000-563ed2fa2000 r--p 000a8000 00:2c 1806417 /var/lib/docker/overlay2/6c6bd4bbe4cb9f0886d24c028d417d9ace5f11afc10964addfbb9ed50947ffde/merged/bin/busybox
...
The issue is also reported in the shiftfs-dkms repo: https://github.com/toby63/shiftfs-dkms/issues/21#issuecomment-1441499170
I doubt this will be fixed as shiftfs is on the way to deprecation (replaced by the Linux kernel's ID-mapped mounts feature since kernel 5.12+).
If you have a host with kernel 5.12+, my suggestion is that you work-around this by removing shiftfs from your host (e.g., sudo rmmod shiftfs) and then restarting Sysbox (sudo systemctl restart sysbox).
If your host is kernel is < 5.12, then you usually need shiftfs do use Sysbox. I say usually because without shiftfs, mounting host files/dirs/volumes into a container won't work properly (e.g., docker run --runtime=sysbox-runc -v /some/host/vol:/mnt ...) because they will show up inside the container as owned by nobody:nogroup. If you are not mounting host file/dirs/volumes into a container, then you can try removing shiftfs and things should work (including running Docker inside the Sysbox container).
FYI, you can also ask Sysbox to not use shiftfs (even if present in the kernel) by configuring the systemd service unit for the sysbox-mgr such that you pass the --disable-shiftfs flag to sysbox-mgr. See the Sysbox docs for more on this.
Hope that helps.