compose build v2 / rootless not working anymore

[pierrehenrymuller]

Description

I have a Fedora 36 uptodate with Docker rootless installation and docker-compose-plugin from official repo. I have a error to build or up with build projects in v2 who works fine with docker-compose 1.X version.

Steps to reproduce the issue: There is a Dockerfile

FROM alpine:3.16
RUN set -x && \
    apk add -u bash

When I made a docker build . I have a correct build of the image :

docker build .
Sending build context to Docker daemon  3.072kB
Step 1/2 : FROM alpine:3.16
3.16: Pulling from library/alpine
213ec9aee27d: Pull complete 
Digest: sha256:bc41182d7ef5ffc53a40b044e725193bc10142a1243f395ee852a8d9730fc2ad
Status: Downloaded newer image for alpine:3.16
 ---> 9c6f07244728
Step 2/2 : RUN set -x &&     apk add -u bash
 ---> Running in 4b209b090373
+ apk add -u bash
fetch https://dl-cdn.alpinelinux.org/alpine/v3.16/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.16/community/x86_64/APKINDEX.tar.gz
(1/4) Installing ncurses-terminfo-base (6.3_p20220521-r0)
(2/4) Installing ncurses-libs (6.3_p20220521-r0)
(3/4) Installing readline (8.1.2-r0)
(4/4) Installing bash (5.1.16-r2)
Executing bash-5.1.16-r2.post-install
Executing busybox-1.35.0-r17.trigger
OK: 8 MiB in 18 packages
Removing intermediate container 4b209b090373
 ---> 40523fc53c7b
Successfully built 40523fc53c7b

When I want to build with compose with this docker-compose.yml :

services:
  test:
    image: test
    build: ./

I have this :

docker compose build 
[+] Building 0.0s (2/2) FINISHED                                                                                                                              
 => [internal] load build definition from Dockerfile                                                                                                     0.0s
 => => transferring dockerfile: 149B                                                                                                                     0.0s
 => [internal] load .dockerignore                                                                                                                        0.0s
 => => transferring context: 2B                                                                                                                          0.0s
failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to read dockerfile: failed to mount /home/phmuller/.local/share/docker/tmp/buildkit-mount2649119911: [{Type:bind Source:/home/phmuller/.local/share/docker/fuse-overlayfs/j1cnjh6vp22ic3wfgmx504g19/diff Options:[rbind ro]}]: operation not permitted

Describe the results you received: I'm not able to build customs images.

Describe the results you expected: Build customs images and start without error.

Output of docker compose version:

Docker Compose version v2.10.2

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  compose: Docker Compose (Docker Inc., v2.10.2)

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 5
 Server Version: 20.10.17
 Storage Driver: fuse-overlayfs
 Logging Driver: json-file
 Cgroup Driver: none
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
 runc version: v1.1.2-0-ga916309f
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  rootless
 Kernel Version: 5.19.8-200.fc36.x86_64
 Operating System: Fedora Linux 36 (Thirty Six)
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 23.11GiB
 Name: jakku
 ID: FXMO:REFX:TRER:I7GC:WOIO:5Z4R:XUTG:YF2I:4O5Z:V6PK:SLMW:GNHT
 Docker Root Dir: /home/phmuller/.local/share/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

WARNING: Running in rootless-mode without cgroups. To enable cgroups in rootless-mode, you need to boot the system in cgroup v2 mode.
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

Additional environment details:

I have this entries in /var/log/messages when I run a docker compose build in a project :

Sep 19 16:36:23 hostname dockerd-rootless.sh[80619]: time="2022-09-19T16:36:23.871864049+02:00" level=info msg="parsed scheme: \"\"" module=grpc
Sep 19 16:36:23 hostname dockerd-rootless.sh[80619]: time="2022-09-19T16:36:23.871897796+02:00" level=info msg="scheme \"\" not registered, fallback to default scheme" module=grpc
Sep 19 16:36:23 hostname dockerd-rootless.sh[80619]: time="2022-09-19T16:36:23.871921799+02:00" level=info msg="ccResolverWrapper: sending update to cc: {[{localhost  <nil> 0 <nil>}] <nil> <nil>}" module=grpc
Sep 19 16:36:23 hostname dockerd-rootless.sh[80619]: time="2022-09-19T16:36:23.871931428+02:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Sep 19 16:36:23 hostname dockerd-rootless.sh[80619]: time="2022-09-19T16:36:23.877955710+02:00" level=warning msg="grpc: addrConn.createTransport failed to connect to {localhost  <nil> 0 <nil>}. Err :connection error: desc = \"transport: Error while dialing only one connection allowed\". Reconnecting..." module=grpc
Sep 19 16:36:23 hostname dockerd-rootless.sh[80619]: time="2022-09-19T16:36:23.883983849+02:00" level=info msg="parsed scheme: \"\"" module=grpc
Sep 19 16:36:23 hostname dockerd-rootless.sh[80619]: time="2022-09-19T16:36:23.884013385+02:00" level=info msg="scheme \"\" not registered, fallback to default scheme" module=grpc
Sep 19 16:36:23 hostname dockerd-rootless.sh[80619]: time="2022-09-19T16:36:23.884023592+02:00" level=info msg="ccResolverWrapper: sending update to cc: {[{localhost  <nil> 0 <nil>}] <nil> <nil>}" module=grpc
Sep 19 16:36:23 hostname dockerd-rootless.sh[80619]: time="2022-09-19T16:36:23.884030715+02:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Sep 19 16:36:23 hostname dockerd-rootless.sh[80619]: time="2022-09-19T16:36:23.889969105+02:00" level=info msg="parsed scheme: \"\"" module=grpc
Sep 19 16:36:23 hostname dockerd-rootless.sh[80619]: time="2022-09-19T16:36:23.890008382+02:00" level=info msg="scheme \"\" not registered, fallback to default scheme" module=grpc
Sep 19 16:36:23 hostname dockerd-rootless.sh[80619]: time="2022-09-19T16:36:23.890041576+02:00" level=info msg="ccResolverWrapper: sending update to cc: {[{localhost  <nil> 0 <nil>}] <nil> <nil>}" module=grpc
Sep 19 16:36:23 hostname dockerd-rootless.sh[80619]: time="2022-09-19T16:36:23.890048369+02:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Sep 19 16:36:23 hostname dockerd-rootless.sh[80619]: time="2022-09-19T16:36:23.895789938+02:00" level=info msg="parsed scheme: \"\"" module=grpc
Sep 19 16:36:23 hostname dockerd-rootless.sh[80619]: time="2022-09-19T16:36:23.895825848+02:00" level=info msg="scheme \"\" not registered, fallback to default scheme" module=grpc
Sep 19 16:36:23 hostname dockerd-rootless.sh[80619]: time="2022-09-19T16:36:23.895840728+02:00" level=info msg="ccResolverWrapper: sending update to cc: {[{localhost  <nil> 0 <nil>}] <nil> <nil>}" module=grpc
Sep 19 16:36:23 hostname dockerd-rootless.sh[80619]: time="2022-09-19T16:36:23.895849331+02:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Sep 19 16:36:24 hostname dockerd-rootless.sh[80619]: time="2022-09-19T16:36:24.097889660+02:00" level=warning msg="grpc: addrConn.createTransport failed to connect to {localhost  <nil> 0 <nil>}. Err :connection error: desc = \"transport: Error while dialing only one connection allowed\". Reconnecting..." module=grpc
Sep 19 16:36:24 hostname dockerd-rootless.sh[80619]: time="2022-09-19T16:36:24.097953029+02:00" level=warning msg="grpc: addrConn.createTransport failed to connect to {localhost  <nil> 0 <nil>}. Err :connection error: desc = \"transport: Error while dialing only one connection allowed\". Reconnecting..." module=grpc
Sep 19 16:36:24 hostname dockerd-rootless.sh[80619]: time="2022-09-19T16:36:24.097973359+02:00" level=warning msg="grpc: Server.processUnaryRPC failed to write status: connection error: desc = \"transport is closing\"" module=grpc
Sep 19 16:36:24 hostname dockerd-rootless.sh[80619]: time="2022-09-19T16:36:24.098057201+02:00" level=warning msg="grpc: Server.processUnaryRPC failed to write status: connection error: desc = \"transport is closing\"" module=grpc
Sep 19 16:36:24 hostname dockerd-rootless.sh[80619]: time="2022-09-19T16:36:24.098179082+02:00" level=warning msg="grpc: addrConn.createTransport failed to connect to {localhost  <nil> 0 <nil>}. Err :connection error: desc = \"transport: Error while dialing only one connection allowed\". Reconnecting..." module=grpc
Sep 19 16:36:24 hostname dockerd-rootless.sh[80619]: time="2022-09-19T16:36:24.116557714+02:00" level=warning msg="grpc: Server.processUnaryRPC failed to write status: connection error: desc = \"transport is closing\"" module=grpc
Sep 19 16:36:24 hostname dockerd-rootless.sh[80619]: time="2022-09-19T16:36:24.125641772+02:00" level=warning msg="grpc: Server.processUnaryRPC failed to write status: connection error: desc = \"transport is closing\"" module=grpc

It's seems that docker compose make too many connection to dockerd-rootless.sh. I don't have find config options to descrease connexion number.

For information when I launch a docker compose up with a compose who don't have build instruction, just an image to launch it's works well.

Do you have an idea about this thing? Thanks in advance

[milas]

The docker build command is not using BuildKit, while Compose is, so likely that's what's not working correctly in the rootless setup.

If you run DOCKER_BUILDKIT=0 docker compose build, does it succeed?

Similarly, if you run DOCKER_BUILDKIT=1 docker build ., does it fail?

[nocive]

Looks like this could be an issue from upstream https://github.com/moby/buildkit/issues/879, for which a PR with a fix already exists https://github.com/moby/buildkit/pull/3097

[joshua1234511]

A quick resolution is to disable "Use Docker Compose V2" (Uncheck it and restart)

[pierrehenrymuller]

@milas If I use DOCKER_BUILDKIT=0 docker compose build for a simple project it's work well. But I have an other problem with API Platform. It doesn't build with the latest docker rootless (20.10.17 latest available for rootless one minus version before main docker version).

@joshua1234511 I'm using docker with CLI in a Linux shell not in docker desktop

[milas]

Similarly, if you run DOCKER_BUILDKIT=1 docker build ., does it fail?

Can you test the above please?

Given that you are able to use Compose without BuildKit, it's not clear to me this is a Compose issue vs a problem with your rootless setup.

[fhaefemeier]

I can reproduce the issue with docker compose V2. I test the following scenario:

• DOCKER_BUILDKIT=0 docker compose build works,
• DOCKER_BUILDKIT=1 docker compose build fails,
• DOCKER_BUILDKIT=0 docker build works.
• DOCKER_BUILDKIT=1 docker build fails.
• docker compose build fails,
• docker build works

[milas]

@fhaefemeier Can you share the output of docker info as well for OS/Engine version?

Given what your repro cases show, which is what I suspected, this is not a Compose-specific issue, but is something with BuildKit and the engine in rootless mode, so we might need to move this issue to another repo.

[ndeloof]

@crazy-max do you have github permissions to transfert issue to docker/buildx ?

[crazy-max]

@ndeloof I don't have perms for transfer but it's BuildKit related.

[ndeloof]

@tonistiigi according to https://github.com/docker/compose/issues/9859#issuecomment-1275177523 this issue is reproducible with a plain docker build with buildkit enable, could you please transfert to buildx repo as I don't have permission to ?