containerd-worker inside container

[pschulten]

I'm trying to run buildkitd with a containerd worker inside a container, but if fails without a clear error message. My container:

# syntax = docker/dockerfile:1.7
FROM debian:unstable

RUN apt-get update && apt-get install -y \
    git containerd supervisor \
    && rm -rf /var/lib/apt/lists/*

COPY --link etc /etc
COPY --link --from=moby/buildkit:latest /usr/bin/buildkitd /usr/bin/buildkitd
COPY --link --from=moby/buildkit:latest /usr/bin/buildctl /usr/bin/buildctl

ENTRYPOINT ["/usr/bin/supervisord"]

my /etc/supervisord.conf:

; this is a comment
[supervisord]
nodaemon=true
logfile=/dev/stdout
logfile_maxbytes=0
logfile_backups=0
loglevel=warn
user=root

[program:containerd]
command=containerd

stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0

[program:buildkitd]
; this works fine
; command=buildkitd --addr=tcp://0.0.0.0:1234 --addr=unix:///run/buildkit/buildkitd.sock

; this fails
command=buildkitd --debug --oci-worker=false --containerd-worker=true --addr=tcp://0.0.0.0:1234 --addr=unix:///run/buildkit/buildkitd.sock

stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0

exec'ing into this container that's running with --privileged and running:

cat <<EOF > Dockerfile
FROM alpine
RUN echo hello
EOF

buildctl build \
  --progress=plain --frontend=dockerfile.v0 --local context=. --local dockerfile=. \
  --output type=image,name=test,push=false

errors with:

....
#4 [1/2] FROM docker.io/library/alpine:latest@sha256:c5b1261d6d3e43071626931fc004f70149baeba2c8ec672bd4f27761f8e1ad6b
#4 resolve docker.io/library/alpine:latest@sha256:c5b1261d6d3e43071626931fc004f70149baeba2c8ec672bd4f27761f8e1ad6b 0.0s done
#4 sha256:bca4290a96390d7a6fc6f2f9929370d06f8dfcacba591c76e3d5c5044e7f420c 0B / 3.35MB 0.2s
#4 sha256:bca4290a96390d7a6fc6f2f9929370d06f8dfcacba591c76e3d5c5044e7f420c 3.35MB / 3.35MB 0.3s done
#4 extracting sha256:bca4290a96390d7a6fc6f2f9929370d06f8dfcacba591c76e3d5c5044e7f420c 0.1s done
#4 DONE 0.4s

#5 [2/2] RUN echo hello
#5 ERROR: process "/bin/sh -c echo hello" did not complete successfully: invalid argument
------
 > [2/2] RUN echo hello:
------
Dockerfile:2
--------------------
   1 |     FROM alpine
   2 | >>> RUN echo hello
   3 |     
--------------------
error: failed to solve: process "/bin/sh -c echo hello" did not complete successfully: invalid argument

Should this work? Am I'm doing something wrong?

[tonistiigi]

I don't see where you are running containerd.

Running containerd and buildkitd in separate containers/rootfs requires quite complicated setup as containerd API is not a remote API but needs the client to share the same filesystem for the mount paths to work.

[pschulten]

Thanks so much @tonistiigi for getting back to me!

They are running both in the same container:

$ docker exec buildkit ps -ef
UID          PID    PPID  C STIME TTY          TIME CMD
root           1       0  0 09:32 ?        00:00:00 /usr/bin/python3 /usr/bin/supervisord
root           7       1  0 09:32 ?        00:00:00 buildkitd --debug --oci-worker=false --containerd-worker=true --addr=tcp://0.0.0.0:1234 --addr=unix:///run/buildkit/buildkitd.sock
root           8       1  0 09:32 ?        00:00:00 containerd
root          90       0  0 09:35 ?        00:00:00 ps -ef

and share the same fs:

$ docker exec buildkit find /run -name *sock*
/run/containerd/containerd.sock.ttrpc
/run/containerd/containerd.sock
/run/buildkit/buildkitd.sock
/run/buildkit/otel-grpc.sock

and I guess that buildkit is actually using the containerd work:

$ docker logs buildkit 2>&1 | grep "org.mobyproject.buildkit.worker.executor:containerd"
time="2024-03-22T09:32:31Z" level=info msg="found worker \"ny8txln84j9y5kuwuut40yooh\", labels=map[org.mobyproject.buildkit.worker.containerd.namespace:buildkit org.mobyproject.buildkit.worker.containerd.uuid:acb9984d-b3d7-4c96-a44c-20ef10b95abf org.mobyproject.buildkit.worker.executor:containerd org.mobyproject.buildkit.worker.hostname:ce53c0968406 org.mobyproject.buildkit.worker.network:host org.mobyproject.buildkit.worker.selinux.enabled:false org.mobyproject.buildkit.worker.snapshotter:overlayfs], platforms=[linux/arm64 linux/amd64 linux/amd64/v2 linux/riscv64 linux/ppc64 linux/ppc64le linux/s390x linux/386 linux/mips64le linux/mips64 linux/loong64]"

If you want to try it out, I published the code here: https://github.com/pschulten/buildkit

Also attached the log output for a failing build docker.out.tar.gz

[kunZooberg]

I am trying same thing, but also failed https://github.com/containerd/containerd/discussions/10237 Do you have any work around?