buildkit
buildkit copied to clipboard
moby
failed to solve (mounting not permitted) when using namespace remapping
I am attempting to utilize docker namespace remapping. Builds are unsuccessful with Dockerfiles that include a syntax directive (Eg # syntax=docker/dockerfile:1.4) with ERROR: failed to solve: exit code: 1.
An example Dockerfile:
#syntax=docker/dockerfile:1.4
FROM mcr.microsoft.com/devcontainers/base
# docker build -t test .
[+] Building 1.1s (5/5) FINISHED docker:default
=> [internal] load build definition from Dockerfile 0.1s
=> => transferring dockerfile: 110B 0.0s
=> [internal] load .dockerignore 0.1s
=> => transferring context: 2B 0.0s
=> resolve image config for docker.io/docker/dockerfile:1.4 0.7s
=> [auth] docker/dockerfile:pull token for registry-1.docker.io 0.0s
=> CACHED docker-image://docker.io/docker/dockerfile:1.4@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc 0.0s
Dockerfile:1
--------------------
1 | >>> # syntax=docker/dockerfile:1.4
2 | FROM mcr.microsoft.com/devcontainers/base
3 |
--------------------
ERROR: failed to solve: exit code: 1
The following 'operation not permitted' error can be found in the journal logs:
Jan 13 00:40:10 f5e30636-1287-4fa2-82c3-c4fee7233074 dockerd[16476]:
runc run failed: unable to start container process: error during container init: error mounting
"/var/lib/docker/231072.231072/tmp/buildkit-metadata1667912507" to rootfs at
"/run/config/buildkit/metadata": mount /proc/self/fd/8:/run/config/buildkit/metadata (via
/proc/self/fd/9), flags: 0x1021: operation not permitted
This repros on a default Ubuntu 22.04 Azure VM, with some limited customizations. One notable customization here is that the /var/lib/docker directory is a mount of an ext4 filesystem (more details hidden below). I don't believe this is the cause of the issue, but thought i'd point it out for added context.
I have no issues building docker images on this host unless combining with user namespace remapping. What i've tried:
â
A Dockerfile without the syntax directive build and WITH user namespace remapping
â
Building Dockerfile WITH syntax directive and WITHOUT user namespace remapping
Â
docker info
$ /bin/sh -c docker info && uname -a
Client:
Version: 24.0.7-1
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: 0.12.0-1
Path: /usr/libexec/docker/cli-plugins/docker-buildx
2024-01-13 00:52:12.136Z: compose: Docker Compose (Docker Inc.)
Version: 2.23.3-1
Path: /usr/libexec/docker/cli-plugins/docker-compose
Server:
Containers: 1
Running: 0
Paused: 0
Stopped: 1
Images: 1
Server Version: 24.0.7-1
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: ********
Using metacopy: false
Native Overlay Diff: false
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 3dd1e886e55dd695541fdcd67420c2888645a495
runc version: ccaecfcbc907d70a7aa870a6650887b901b25b82
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
userns
cgroupns
Kernel Version: 6.2.0-1018-azure
Operating System: Ubuntu 22.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.812GiB
Name: 8c3ef404-b5db-4e2a-ac6b-b36b51343d9b
ID: 0c258cbb-762f-4ba7-972b-6dedb9d9ce83
Docker Root Dir: /var/lib/docker/231072.231072
Debug Mode: false
Username: codespacesdev
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
/etc/docker/daemon.json
{
"userns-remap": "default"
}
mount
/dev/sdb1 on / type ext4 (rw,relatime,discard,errors=remount-ro)
devtmpfs on /dev type devtmpfs (rw,nosuid,noexec,relatime,size=1996516k,nr_inodes=499129,mode=755,inode64)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,size=799536k,nr_inodes=819200,mode=755,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13938)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
ramfs on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
/var/lib/snapd/snaps/core20_2015.snap on /snap/core20/2015 type squashfs (ro,nodev,relatime,errors=continue,threads=single,x-gdu.hide)
/var/lib/snapd/snaps/lxd_24322.snap on /snap/lxd/24322 type squashfs (ro,nodev,relatime,errors=continue,threads=single,x-gdu.hide)
/var/lib/snapd/snaps/snapd_20290.snap on /snap/snapd/20290 type squashfs (ro,nodev,relatime,errors=continue,threads=single,x-gdu.hide)
/dev/sdb15 on /boot/efi type vfat (rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
/dev/sda1 on /mnt type ext4 (rw,relatime,x-systemd.requires=cloud-init.service,_netdev)
tmpfs on /run/snapd/ns type tmpfs (rw,nosuid,nodev,size=799536k,nr_inodes=819200,mode=755,inode64)
nsfs on /run/snapd/ns/lxd.mnt type nsfs (rw)
/var/lib/snapd/snaps/snapd_20671.snap on /snap/snapd/20671 type squashfs (ro,nodev,relatime,errors=continue,threads=single,x-gdu.hide)
/var/lib/snapd/snaps/core20_2105.snap on /snap/core20/2105 type squashfs (ro,nodev,relatime,errors=continue,threads=single,x-gdu.hide)
//vsocef459ab970a3b4b28b5f.z21.file.storage.azure.net/cloudenvdata on /mnt/cloudenvdata type cifs (rw,relatime,vers=3.1.1,cache=strict,username=vsocef459ab970a3b4b28b5f,uid=0,noforceuid,gid=0,noforcegid,addr=52.239.237.8,file_mode=0777,dir_mode=0777,soft,persistenthandles,nounix,serverino,mapposix,rsize=1048576,wsize=1048576,bsize=1048576,echo_interval=60,actimeo=1,closetimeo=1)
/dev/loop5 on /var/lib/docker type ext4 (rw,nodev,relatime)
/dev/loop6 on /var/log/codespaces/backup type ext4 (rw,nodev,relatime)
OS/kernel info
# cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
# uname -a
Linux 6e4d4f2d-4af8-4041-95b3-06a832ca6b35 6.2.0-1018-azure #18~22.04.1-Ubuntu SMP Tue Nov 21 19:25:02 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Repro in codespaces
Edit: This also repros easily in a codespace with the following steps:
- Create a GitHub Codespace on nearly any repo (can't have a custom devcontainer.json unless it also includes docker. I've tested with the official
docker-in-dockerv2.7.1 Feature).- Using this repo
moby/buildkitworks fine for the repro.
- Using this repo
- Enable user namespace remapping in
/etc/docker/daemon.json. - Kill and restart dockerd (
pkill dockerd && dockerd). - Try to build the Dockerfile above
docker info
@joshspicer â /workspaces/buildkit (master) $ docker info && uname -a
Client:
Version: 24.0.7-1
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: 0.12.0-1
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: 2.23.3-1
Path: /usr/libexec/docker/cli-plugins/docker-compose
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 24.0.7-1
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: false
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 3dd1e886e55dd695541fdcd67420c2888645a495
runc version: 4bccb38cc9cf198d52bebf2b3a90cd14e7af8c06
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
userns
cgroupns
Kernel Version: 6.2.0-1018-azure
Operating System: Ubuntu 20.04.6 LTS (containerized)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 7.744GiB
Name: codespaces-7372cd
ID: 65629d15-48c3-4cf1-adef-23a3c0da7b2a
Docker Root Dir: /var/lib/docker/165536.165536
Debug Mode: false
Username: codespacesdev
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Linux codespaces-7372cd 6.2.0-1018-azure #18~22.04.1-Ubuntu SMP Tue Nov 21 19:25:02 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
The only similar discussion i've found online thus far is https://github.com/containerd/nerdctl/discussions/1401 .
There are a non-zero number people in our userbase that seem to rely on providing the syntax directive (https://github.com/microsoft/vscode-remote-release/issues/7463) so just stripping it from Dockerfiles doesn't seem like an option (and is probably hiding a deeper issue)
Possibly similar error found in the rootless docs: https://github.com/moby/buildkit/blob/master/docs/rootless.md#error-mount-procproc-via-procselffd6-flags-0xe-operation-not-permitted
I do notice that the tmp directory under the remapped user's /var/lib/docker is not in the same group as some of the other directories. I've tried creating this and manually changing ownership and updating permissions with the sticky bit (g+rwxs), but it didn't seem to help in this case
@tonistiigi Do you perhaps have any quick suggestions I could try to get around this issue? 🙏
It seems to be an issue with this mount https://github.com/moby/buildkit/blob/master/frontend/gateway/gateway.go#L341-L348 not converting the path to file accessible for the remapped user. worker.CacheManger().IdentityMapping() should give the parameters needed fix the permissions.
FWIW i've found a much easier way to repro in Codespaces. I've edited the issue description above (see the Repro in codespaces section)
I think some solution similar to https://github.com/moby/buildkit/issues/3098 may help here.
Any tips on running/simulating a development builder under a user namespace? I see the buildkitd --rootless flag but i'm not yet able to repro this issue compiling from source. I'd love to get a repro building from source so I could test a potential fix (cc/ @tonistiigi if you have a doc or quick suggestion here). Is running buildkitd in a linux username enough to simulate the user-ns docker flag?
This indeed produces in codespaces but not in any other system I've tested. Also does not produce in dind. I thought that this was maybe apparmor, but even in systems with apparmor enabled I see no issue. As codespaces does not use official Docker release (and I don't know how the VM is set up) can't debug this more before additional info is provided.