[rootlesskit:parent] error: failed to setup UID/GID map [...] newuidmap: write to uid_map failed: Operation not permitted

[yuvipanda]

Awesome project :) I'm trying to use rootlesskit directly, since I want isolation without necessarily image management, etc.

I'm trying to run it inside a docker container, with this Dockerfile:

FROM ubuntu:23.10

RUN apt update > /dev/null && apt install --yes rootlesskit

RUN useradd test-user

USER test-user

And I build this image, and run it with docker run -it --privileged --security-opt seccomp=unconfined --security-opt apparmor=unconfined wat /bin/bash.

But rootlesskit just fails, with

test-user@a674e50fe5f3:/$ rootlesskit --debug bash
DEBU[0000] subid-source: using the dynamic source       
DEBU[0000] Executing [/usr/bin/getsubids test-user]     
DEBU[0000] Executing [/usr/bin/getsubids 1001]          
DEBU[0000] reaper: auto chosen value: false             
DEBU[0000] subid-source:dynamic: failed to get subuids by the UID 1001  error="failed to exec [/usr/bin/getsubids 1001]: exit status 1 (stdout=\"\", stderr=\"Error fetching ranges\\n\")"
DEBU[0000] Executing [/usr/bin/getsubids -g test-user]  
DEBU[0000] Executing [/usr/bin/getsubids -g 1001]       
DEBU[0000] subid-source:dynamic: failed to get subgids by the UID 1001  error="failed to exec [/usr/bin/getsubids -g 1001]: exit status 1 (stdout=\"\", stderr=\"Error fetching ranges\\n\")"
DEBU[0000] subuid ranges=[{165536 65536}]               
DEBU[0000] subgid ranges=[{165536 65536}]               
[rootlesskit:parent] error: failed to setup UID/GID map: newuidmap 14 [0 1001 1 1 165536 65536] failed: newuidmap: write to uid_map failed: Operation not permitted
: exit status 1
[rootlesskit:child ] error: parsing message from fd 3: EOF

/etc/subuid and /etc/subgid look fine:

ubuntu:100000:65536
test-user:165536:65536

Dropping the --privileged flag seems to have no effect either.

[yuvipanda]

I also tried getcap /usr/bin/newuidmap, but that returned nothing. So I added the following lines to my Dockerfile:

RUN setcap cap_setuid=ep /usr/bin/newuidmap
RUN setcap cap_setuid=ep /usr/bin/newgidmap

This does seem to set the capability correctly, as getcap now returns /usr/bin/newuidmap cap_setuid=ep. However, I'm not 100% sure this is necessary, and regardless, rootleskit --debug bash still produces the same output

[AkihiroSuda]

What's your host OS and docker version?

The dockerfile works for me with Docker 24.0.7 on Ubuntu 22.04.3 LTS (kernel 5.15.0-86-generic).

[AkihiroSuda]

• If you are trying to rootlesskit inside rootless docker, you'll have to increase the number of the subids on the host
• Does rootlesskit work on the host?
• You may need some sysctl, depending on the host distro https://rootlesscontaine.rs/getting-started/common/sysctl/ https://ubuntu.com/blog/ubuntu-23-10-restricted-unprivileged-user-namespaces

[yuvipanda]

Just getting back to this, sorry for the really late response.

Here's the docker info:

Client: Docker Engine - Community
 Version:    25.0.3
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.12.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.24.5
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 1
 Server Version: 25.0.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.0-92-generic
 Operating System: Ubuntu 22.04.3 LTS
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 11.66GiB
 Name: lima-aarch64-rootful
 ID: ed205f0b-98e2-4221-98d1-ef8609f92896
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

I'm not trying to run this inside rootless docker - just regular docker.

This is Ubuntu 22.04, with kernel 5.15.0-92-generic on aarch64. From what I can tell reading the two links, it doesn't need any sysctl setup - it's too old for the 23.10 ntroduced userns but new enough to need kernel.unprivileged_userns_clone=1 explicitly (I tried it anyway to no effect).

rootlesskit does work on the host.

[zhoufenqin]

@AkihiroSuda same error in Debain Bulleye, the Dockerfile like this:

ARG BASE_BUILDER_IMAGE="mcr.microsoft.com/oryx/builder:debian-bullseye-20240124.1"
FROM ${BASE_BUILDER_IMAGE}

USER root

# download and install file package
RUN set -eux; \
	apt-get update; \
	apt-get install -y --no-install-recommends \
		rootlesskit \ 
	; \
	rm -rf /var/lib/apt/lists/*


ENV USER cnb
ENTRYPOINT ["rootlesskit", "sh"]

error shows:

~/code/github/aca-source-to-cloud-builder/aca-builder (fenzho/buildkit ✗) docker run -it --rm --user=1000 test1       
[rootlesskit:parent] error: failed to setup UID/GID map: newuidmap 11 [0 1000 1 1 100000 65536] failed: newuidmap: write to uid_map failed: Operation not permitted
: exit status 1

If I apt-get install uidmap, then download rootlesskit package manually, it still not work

ARG BASE_BUILDER_IMAGE="mcr.microsoft.com/oryx/builder:debian-bullseye-20240124.1"
FROM ${BASE_BUILDER_IMAGE}

USER root

# download and install file package
RUN set -eux; \
	apt-get update; \
	apt-get install -y --no-install-recommends \
	uidmap \
	; \
	rm -rf /var/lib/apt/lists/*

RUN curl -SsLf "https://github.com/rootless-containers/rootlesskit/releases/download/v1.0.1/rootlesskit-x86_64.tar.gz" | tar Cxzv /usr/bin

ENV USER cnb
ENTRYPOINT ["rootlesskit", "sh"]

If I install the newuidmap and newgidmap from shadow, then it works

ARG BASE_BUILDER_IMAGE="mcr.microsoft.com/oryx/builder:debian-bullseye-20240124.1"
FROM ${BASE_BUILDER_IMAGE}

USER root

# download and install file package
RUN set -eux; \
	apt-get update; \
	apt-get install -y --no-install-recommends \
        git autoconf autopoint libtool automake make bison gettext gcc libcap-dev  \
	; \
	rm -rf /var/lib/apt/lists/*

RUN curl -SsLf "https://github.com/rootless-containers/rootlesskit/releases/download/v1.0.1/rootlesskit-x86_64.tar.gz" | tar Cxzv /usr/bin

ARG SHADOW_VERSION=4.8.1
RUN git clone https://github.com/shadow-maint/shadow.git /shadow
WORKDIR /shadow
ARG SHADOW_VERSION
RUN git pull && git checkout $SHADOW_VERSION
RUN ./autogen.sh --disable-nls --disable-man --without-audit --without-selinux --without-acl --without-attr --without-tcb --without-nscd && \
  make && \
  cp src/newuidmap src/newgidmap /usr/bin

#https://github.com/moby/moby/issues/41812
RUN chmod u+s /usr/bin/newuidmap
RUN chmod u+s /usr/bin/newgidmap
ENV USER cnb
ENTRYPOINT ["rootlesskit", "sh"]

why rootlesskit doesn't work in Debain?

[zhoufenqin]

It may because Debian doesn't set cap for /usr/bin/newuidmap and /usr/bin/newgidmap, it works after setting the capabilities:

RUN chmod 0755 /usr/bin/newuidmap /usr/bin/newgidmap && \
    setcap cap_setuid=ep /usr/bin/newuidmap && \
    setcap cap_setgid=ep /usr/bin/newgidmap

Example:

ARG BASE_BUILDER_IMAGE="mcr.microsoft.com/oryx/builder:debian-bullseye-20240124.1"
FROM ${BASE_BUILDER_IMAGE}

USER root

# download and install file package
RUN set -eux; \
	apt-get update; \
	apt-get install -y --no-install-recommends \
	uidmap libcap2-bin\
	; \
	rm -rf /var/lib/apt/lists/*

RUN chmod 0755 /usr/bin/newuidmap /usr/bin/newgidmap && \
    setcap cap_setuid=ep /usr/bin/newuidmap && \
    setcap cap_setgid=ep /usr/bin/newgidmap

RUN curl -SsLf "https://github.com/rootless-containers/rootlesskit/releases/download/v1.0.1/rootlesskit-x86_64.tar.gz" | tar Cxzv /usr/bin

ENV USER cnb
ENTRYPOINT ["rootlesskit", "sh"]

[yuvipanda]

That's really interesting @zhoufenqin! I had the following lines in my Dockerfile:

RUN setcap cap_setuid=ep /usr/bin/newuidmap
RUN setcap cap_setuid=ep /usr/bin/newgidmap

but that doesn't work.

But

RUN chmod 0755 /usr/bin/newuidmap /usr/bin/newgidmap && \
    setcap cap_setuid=ep /usr/bin/newuidmap && \
    setcap cap_setgid=ep /usr/bin/newgidmap

does?!

[yuvipanda]

Aaaaah, it's because my second one was doing cap_setuid when it should have been cap_setgid 🤦🏾‍♂️