rootless-containers
Encountered error `[rootlesskit:parent] error: failed to start the child: fork/exec /proc/self/exe: operation not permitted` when build the docker image
Hi, I'm using rootless docker for use Docker in Docker (DinD). I need a hand with an error like the titile.
What happened?
We are using the following Dockerfile (and entrypoint), but the build has not been going through in the last week or so. (In other words, the build was working correctly until last week)
FROM ubuntu:22.04
RUN apt-get update && apt-get install -y --no-install-recommends \
kmod \
dbus-user-session \
curl \
ca-certificates \
uidmap \
openssl \
gnupg \
gpg \
iproute2 \
&& rm -rf /var/lib/apt/lists/* \
&& rm -rf /tmp/*
RUN install -m 0755 -d /etc/apt/keyrings; \
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg; \
chmod a+r /etc/apt/keyrings/docker.gpg; \
echo \
"deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
"$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
tee /etc/apt/sources.list.d/docker.list > /dev/null
RUN apt-get update && apt-get upgrade -y; \
apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin docker-ce-rootless-extras
RUN echo "kernel.unprivileged_userns_clone=1" >> /etc/sysctl.conf; \
echo "user.max_user_namespaces=28633" >> /etc/sysctl.conf; \
sysctl --system
RUN set -eux; \
addgroup --system dockremap; \
adduser --system --no-create-home --ingroup=dockremap dockremap; \
echo 'dockremap:165536:65536' >> /etc/subuid; \
echo 'dockremap:165536:65536' >> /etc/subgid
RUN set -eux; \
addgroup --system docker; \
addgroup --system --gid=1000 rootless; \
adduser --disabled-password --uid=1000 --ingroup=rootless rootless; \
adduser rootless docker
COPY rootless_entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh
RUN mkdir -p /home/rootless/.local/share/docker
RUN chown -R rootless:rootless /home/rootless/.local/share/docker
RUN mkdir /certs /certs/client && chmod 1777 /certs /certs/client
USER rootless
RUN dockerd-rootless-setuptool.sh install --skip-iptables
CMD ["/entrypoint.sh"]
The rootless_entrypoint.sh is below.
#!/bin/bash
echo "Generating Docker TLS certificate and key..."
mkdir -p /certs
openssl genrsa -out /certs/key.pem 4096
openssl req -new -x509 -sha256 -key /certs/key.pem -out /certs/cert.pem -days 365 -subj '/CN=localhost'
echo "Starting Docker..."
dockerd-rootless.sh -H ${DOCKER_HOST} --tlsverify --tlscacert=/certs/cert.pem --tlscert=/certs/cert.pem --tlskey=/certs/key.pem
The error message is as follows:
$ ls
Dockerfile rootless_entrypoint.sh
$ docker build -t rootless-docker .
=> ERROR [13/13] RUN dockerd-rootless-setuptool.sh install --skip-iptables 0.1s
------
> [13/13] RUN dockerd-rootless-setuptool.sh install --skip-iptables:
0.104 [rootlesskit:parent] error: failed to start the child: fork/exec /proc/self/exe: operation not permitted
0.104 [ERROR] RootlessKit failed, see the error messages and https://rootlesscontaine.rs/getting-started/common/ .
------
Dockerfile:49
--------------------
47 | RUN mkdir /certs /certs/client && chmod 1777 /certs /certs/client
48 | USER rootless
49 | >>> RUN dockerd-rootless-setuptool.sh install --skip-iptables
50 |
51 | CMD ["/entrypoint.sh"]
--------------------
ERROR: failed to solve: process "/bin/sh -c dockerd-rootless-setuptool.sh install --skip-iptables" did not complete successfully: exit code: 1
Environment
Build machine: Mac Studio Apple M2 Ultra OS: macOS 13.6 Docker: Docker version 25.0.3, build 4debf41
What do I do to solve this problem?
Commonly, this problem ocurre when the value of /proc/sys/kernel/unprivileged_userns_clone is set to 0.
https://docs.docker.com/engine/security/rootless/#errors-when-starting-the-docker-daemon
But I did set the kernel.unprivileged_userns_clone=1 to /etc/sysct.conf in Dockerfile.
I am facing exactly the same issue trying to build rootless dind container (with Docker version 24.0.9, build 2936816) on Github Actions (ubuntu-22.04) runners using debian-stable base image.
Things were working fine until beginning of last week, now suddenly same build scripts fail with this error.
Running sudo sh -c "echo 1 > /proc/sys/kernel/unprivileged_userns_clone" or sudo sysctl -w kernel.unprivileged_userns_clone=1 before the docker build on the host system has no effect. the error persists.
I tried different versions of rootlesskit (2.0.2, 2.0.1, 1.1.1) all fail now with [rootlesskit:parent] error: failed to start the child: fork/exec /proc/self/exe: operation not permitted despite kernel.unprivileged_userns_clone=1.
I also read that AppArmor can be the reason for the error: https://discuss.linuxcontainers.org/t/rootless-docker-on-new-ubuntu-kernels-does-not-work/18708 However when I disabled and even when I removed AppArmor the build still fails with [rootlesskit:parent] error: failed to start the child: fork/exec /proc/self/exe: operation not permitted
@sebthom Thank you for your response.
I tried different versions of rootlesskit (2.0.2, 2.0.1, 1.1.1) all fail now with [rootlesskit:parent] error: failed to start the child: fork/exec /proc/self/exe: operation not permitted despite kernel.unprivileged_userns_clone=1.
I reproduced the same behavior when I tried docker build use with rootlesskit (v2.0.2, v2.0.1).
Upon investigation I noticed that a new version of docker-ce-rootless-extras:5:25.0.4-1~ubuntu.22.04~jammy has been released.
The build was successful with the following changes to the Dockerfile I initially presented.
diff --git a/rootless-docker/Dockerfile b/rootless-docker/Dockerfile
index be8105d..fc7cacd 100644
--- a/rootless-docker/Dockerfile
+++ b/rootless-docker/Dockerfile
@@ -30,7 +30,9 @@ RUN install -m 0755 -d /etc/apt/keyrings; \
"$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
tee /etc/apt/sources.list.d/docker.list > /dev/null
RUN apt-get update && apt-get upgrade -y; \
- apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin docker-ce-rootless-extras
+ apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
+
+RUN apt-get install -y docker-ce-rootless-extras=5:25.0.3-1~ubuntu.22.04~jammy --allow-downgrades
rootlesskit version is v2.0.1.
$ docker build --no-cache -f Dockerfile --progress=plain -t rootless-docker-test .
$ docker run --rm -it rootless-docker-test rootlesskit --version
rootlesskit version 2.0.1
I assume from the above that there has been some change in package of docker-ce-rootless-extras. However, it is not yet known which dependencies were changed.
Pinning the previous version also works on Debian apt-get install --no-install-recommends -y docker-ce-rootless-extras=5:25.0.3-1~debian.12~bookworm --allow-downgrades.
Thanks for the workaround!
Yes, I would also try to avoid the error by this workaround. But I don't want to keep using an old version, so I need to investigate the root case.
This might be related to AppArmor (at least in my case)
Encountered the same when running containerd-rootless-setuptool.sh check and rootlesskit bash
so need to add an exclusion for AppArmor service (stopping/disabling the service didnt worked out for me)
########## BEGIN ##########
cat <<EOT | sudo tee "/etc/apparmor.d/home.ubuntu.bin.rootlesskit"
# ref: https://ubuntu.com/blog/ubuntu-23-10-restricted-unprivileged-user-namespaces
abi <abi/4.0>,
include <tunables/global>
/home/ubuntu/bin/rootlesskit flags=(unconfined) {
userns,
# Site-specific additions and overrides. See local/README for details.
include if exists <local/home.ubuntu.bin.rootlesskit>
}
EOT
# ref: https://ubuntu.com/blog/ubuntu-23-10-restricted-unprivileged-user-namespaces
abi <abi/4.0>,
include <tunables/global>
/home/ubuntu/bin/rootlesskit flags=(unconfined) {
userns,
# Site-specific additions and overrides. See local/README for details.
include if exists <local/home.ubuntu.bin.rootlesskit>
}
########## END ##########
We are also seeing this issue. Started seeing it on March 7th shortly after 2.0.2 was released.
It looks like 2.0.2 added a check that rootlesskit works to dockerd-rootless-setuptool.sh: https://github.com/moby/moby/blob/b32cfc3b3a9d791ed3924ab168f27962dceec949/contrib/dockerd-rootless-setuptool.sh#L273C2-L278C4. Is it possible that rootlesskit can't run successfully outside a privileged container, e.g. in a docker build?
dockerd-rootless-setuptool.sh install is designed for systemd environments and it does not make sense to run the script inside Dockerfile.
@DenisBalan's suggestion fixed it for me, but now I have a different error presumably needing a different AppArmor permission?
time="2024-09-03T13:20:34Z" level=warning msg="failed to get disk size: no such file or directory"
time="2024-09-03T13:20:34Z" level=warning msg="failed to get disk size: no such file or directory"
buildkitd: mkdir /var/run/user/1001: permission denied
[rootlesskit:child ] error: command [buildkitd --addr=unix:///var/run/user/1001/buildkit/rootless.sock] exited: exit status 1
[rootlesskit:parent] error: child exited: exit status 1
Is there a complete list somewhere @AkihiroSuda (I see you authored the hint Denis linked to above)?
This is running:
socket="unix:///var/run/user/$(id -u)/buildkit/rootless.sock"
rootlesskit --net=slirp4netns --copy-up=/etc --disable-host-loopback buildkitd --addr="$socket" &
This worked fine in Github's ubuntu-24.04 runners, but with 'Ubuntu 24.04 by Arm Limited' for a larger (and aarch64) runner it needed the app armor policy above and slirp4netns & uidmap explicitly installed to get as far as the error above.
