compose
compose copied to clipboard
docker
Referencing local image name fails on "docker compose build"
Description
I'm getting an error from docker compose build with a setup that is consistently successful with docker-compose build.
I have a docker-compose.yml file with two services: base and extended. base.image gives a name to the image built by base, and I'd like to use that name as the FROM image in the Dockerfile for the extended service.
This works well with docker-compose build. It does not work with docker compose build.
Steps to reproduce the issue:
- Re-create my
docker/folder with these three files, plus an emptyREADME.md.cdintodocker/.
# docker/docker-compose.yml
services:
base:
image: neilyio/base
build:
context: .
dockerfile: base.Dockerfile
extended:
build:
context: .
dockerfile: extended.Dockerfile
# docker/base.Dockerfile
FROM scratch
COPY README.md /root/README.md
# docker/extended.Dockerfile
FROM neilyio/base
CMD cat /root/README.md
docker-compose build, expect a successful run.- Clear your cache and delete these new images so we have a clean comparison for the next step. I used these commands:
docker image rm neilyio/base docker_extended
docker system prune -f
docker compose build, expect a failure.
Describe the results you received:
docker compose build produces:
[+] Building 0.6s (8/8) FINISHED
=> [docker_extended internal] load build definition from extended.Dockerfile 0.0s
=> => transferring dockerfile: 88B 0.0s
=> [neilyio/base internal] load build definition from base.Dockerfile 0.0s
=> => transferring dockerfile: 86B 0.0s
=> [docker_extended internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [neilyio/base internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> ERROR [docker_extended internal] load metadata for docker.io/neilyio/base:latest 0.4s
=> [neilyio/base internal] load build context 0.0s
=> => transferring context: 3.65kB 0.0s
=> [neilyio/base 1/1] COPY README.md /root/README.md 0.0s
=> [neilyio/base] exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:e29ad2347eed9046148ed435ca66984c9421c1d39ae9f40004a62658e60640c3 0.0s
=> => naming to docker.io/neilyio/base 0.0s
------
> [docker_extended internal] load metadata for docker.io/neilyio/base:latest:
------
failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to create LLB definition: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Describe the results you expected:
I expected docker compose build to find the locally-built neilyio/base image, instead it seems to try and load from docker.io/neilyio/base:latest. I expected docker compose build to have the same behaviour as docker-compose build, which successfully found the local image.
Additional information you deem important (e.g. issue happens only occasionally):
This can be a little tricky to reproduce because of Docker's caching. docker compose build will work fine if neilyio/base is already built. docker compose build will successfully find the local image, so it can give the impression that it's working. My step 3 above, clearing the cache, is important to accurately reproduce this. I found I needed to do both a system prune and image rm for this.
docker-compose build works every time, whether or not neilyio/base has been built before.
Output of docker version:
Client:
Cloud integration: 1.0.14
Version: 20.10.6
API version: 1.41
Go version: go1.16.3
Git commit: 370c289
Built: Fri Apr 9 22:46:57 2021
OS/Arch: darwin/arm64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.6
API version: 1.41 (minimum version 1.12)
Go version: go1.13.15
Git commit: 8728dd2
Built: Fri Apr 9 22:44:13 2021
OS/Arch: linux/arm64
Experimental: false
containerd:
Version: 1.4.4
GitCommit: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
runc:
Version: 1.0.0-rc93
GitCommit: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
docker-init:
Version: 0.19.0
GitCommit: de40ad0
Output of docker context show:
default
Output of docker info:
Client:
Context: default
Debug Mode: false
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
compose: Docker Compose (Docker Inc., 2.0.0-beta.1)
scan: Docker Scan (Docker Inc., v0.8.0)
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 2
Server Version: 20.10.6
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
init version: de40ad0
Security Options:
seccomp
Profile: default
Kernel Version: 5.10.25-linuxkit
Operating System: Docker Desktop
OSType: linux
Architecture: aarch64
CPUs: 4
Total Memory: 1.942GiB
Name: docker-desktop
ID: OP3D:IHZS:FQCX:56ZP:HNOA:X4KO:2EF2:AOY2:URIC:5GF6:LUHX:Z7QD
Docker Root Dir: /var/lib/docker
Debug Mode: false
HTTP Proxy: http.docker.internal:3128
HTTPS Proxy: http.docker.internal:3128
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Additional environment details (AWS ECS, Azure ACI, local, etc.):
Local run on M1 Macbook Air.
I realize this is probably because of the parallelism that is part of the compose-cli project, but this seems to be a case where that parallelism is getting in the way.
It seems to me this indeed only works "by chance" as docker-compose build run sequentially, building first image with a tag, then using it as base image for the next service.
I'm not sure if/how we can support such a use-case, would need to check how buildkit can handle this.
I'm facing this issue too. Is it possible to analyze the build dependency by marking possibly used as base image against the images which specify both image and build? Do we need a new option to specify build dependency in the compose config?
Having this issue as well. I tried defining depends_on: on the other services but still didn't work. Went back to use docker-compose (2nd time I had to bail on using the new command)
Do we need a new option to specify build dependency in the compose config?
IMO this is what depends_on: was being used for previously. Docker Compose V1 was smart enough to see that an image in depends_on: didn't exist and that when defined with local build configuration in docker-compose.yml, to build that image first before being used.
A lot of people depend on this behaviour being the same when moving to V2.
A workaround I see is manually building the images in depends_on: in the docker-compose.yml first before building the rest of the images. This is a large maintenance burden though as developers would need to keep track of these image names in a static configuration/scripts at the very least. For this reason, I'm holding off on V2.
@ndeloof This issue is breaking behaviour in a lot of repos I'm seeing and should be addressed.
Same issue applies to compose v1 as long as buildkit is enabled: https://github.com/docker/compose/issues/8449
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
![stale[bot] avatar](https://avatars.githubusercontent.com/in/1724?v=4 "Published by a pub.dev verified publisher"){kind=small}
This issue has been automatically marked as not stale anymore due to the recent activity.
Workarounds depend a bit on your exact use-case;
You can run the builds for each service manually to make sure to build the base image first (docker compose build base), but this depends on what "builder" you use; as it won't work if you use a remote or "container" builder (such builders store build-cache, but not images).
The other workaround (this would usually be the recommended approach) is to use a multi-stage build. However, this assumes the situation as outlined in this ticket's description where both images share the same build-context.
Rewriting the example to have both services use the same Dockerfile, but a different target (stage). The second (extended) stage depends on the first (base) stage, which means that building extended will also build the base stage.
# docker/docker-compose.yml
services:
base:
image: neilyio/base
build:
context: .
target: base
extended:
build:
context: .
target: extended
# syntax=docker/dockerfile:1
FROM scratch AS base
COPY README.md /root/README.md
FROM base AS extended
CMD cat /root/README.md
It's worth noting that;
When building only extended, the layer(s) for base will be built, but no image (neilyio/base) is tagged for the base image.
docker compose build extended
[+] Building 2.9s (9/9) FINISHED
=> [internal] load build definition from Dockerfile 0.2s
=> => transferring dockerfile: 32B 0.0s
=> [internal] load .dockerignore 0.3s
=> => transferring context: 2B 0.0s
=> resolve image config for docker.io/docker/dockerfile:1 2.0s
=> CACHED docker-image://docker.io/docker/dockerfile:1@sha256:443aab4ca21183e069e7d8b2dc68006594f40bddf1b15bbd83f5137bd93e80e2 0.0s
=> [internal] load build definition from Dockerfile 0.0s
=> [internal] load .dockerignore 0.0s
=> [internal] load build context 0.1s
=> => transferring context: 30B 0.0s
=> CACHED [base 1/1] COPY README.md /root/README.md 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:ee813e9db95b2b2d222f4cf72b4507ece85773201a26634f5066ff7d2d4bec65 0.0s
=> => naming to docker.io/library/compose-multibuild_extended 0.0s
docker image ls --filter reference=neilyio/base
REPOSITORY TAG IMAGE ID CREATED SIZE
docker compose build base
[+] Building 1.4s (9/9) FINISHED
=> [internal] load build definition from Dockerfile 0.1s
=> => transferring dockerfile: 32B 0.0s
=> [internal] load .dockerignore 0.1s
=> => transferring context: 2B 0.0s
=> resolve image config for docker.io/docker/dockerfile:1 0.8s
=> CACHED docker-image://docker.io/docker/dockerfile:1@sha256:443aab4ca21183e069e7d8b2dc68006594f40bddf1b15bbd83f5137bd93e80e2 0.0s
=> [internal] load build definition from Dockerfile 0.0s
=> [internal] load .dockerignore 0.0s
=> [internal] load build context 0.1s
=> => transferring context: 30B 0.0s
=> CACHED [base 1/1] COPY README.md /root/README.md 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:67e08e818eb01fd3b97ad637b5e99762d34c766e2bb11f2c501a716ef183de62 0.0s
=> => naming to docker.io/neilyio/base 0.0s
Generally, building extended, then building base should produce the same image (common layers shared between both images), but there may be some improvements to be made in compose here; if both builds would run in parallel, they're not guaranteed to produce the same layer-digests (if changes are made in between within the build-context).
docker inspect --format '{{json .RootFS.Layers}}' neilyio/base
["sha256:3b1be5298307e60029517bf38caa3f1b58121ef39d56d832b4a62dbabc56c3d3"]
docker inspect --format '{{json .RootFS.Layers}}' compose-multibuild_extended
["sha256:3b1be5298307e60029517bf38caa3f1b58121ef39d56d832b4a62dbabc56c3d3"]
@thaJeztah thank you for the detailed response. From what I've been reading online, it was my conclusion too.
My situation is tad more dependency chain problematic, where I'm building a base and using that local image in 2 other local images (e.g. foo and bar) which are using FROM base. I suppose adopting your approach would mean quite some changes to the file system structure (all images are built from root/base, root/foo, root/bar, etc. currently). Also, when I looked at this earlier, it lead me to cache-from and target in docker-compose.yml and I was really hoping to avoid all that.
Any idea on ETA / priority for this issue?
Thanks again.
@curlybeast Recent docker compose release will build images with respect to the depends_on directive, so that you can declare the base image to be built first, then the other service images.
Not sure if this valid or not, but I have same issue and I solve it by using this 2 commands :
export DOCKER_BUILDKIT=0
export COMPOSE_DOCKER_CLI_BUILD=0
After i run that I re-run docker-compose up/build and its solved.
@curlybeast Recent docker compose release will build images with respect to the
depends_ondirective, so that you can declare thebaseimage to be built first, then the other service images.
Thanks for letting me know about this. Which version is this released in?
for me this is fixed when i start use docker compose instead of docker-compose. because docker-compose outdated.
for me this is fixed when i start use
docker composeinstead ofdocker-compose. becausedocker-composeoutdated.
Just be careful you don't have another version in your PATH. I just realised I had an old docker-compose in /usr/local/bin from back in the days when it wasn't installed as standard with docker packages. This is why I couldn't see any of the new features at first :roll_eyes: (like docker compose pull --ignore-pull-failures)
To better cover this scenario, it seems to be we should define a new depends_on condition dedicated to build requirements, i.e.
services:
base:
build: .
extened:
build:
context: extended
depends_on:
base:
condition: image_built
I'll experiment with this approach and prepare a proposal on https://github.com/compose-spec/compose-spec
Any updates on this issue? It seems like docker compose v2.18.1 doesn't respect depends_on for the build when build kit is enabled.
@pebo can you provide a simple example to reproduce this issue?
Docker Compose run builds in depends_on order to address this need (while dependency might not be required at runtime, so my comment)
@ndeloof It seems like the problem arises when the dependent service is using the image property rather than a build section.
Dockerfile
FROM alpine as base
RUN echo 'foo' > /foo.txt
RUN echo 'sleeping for 10s' && sleep 10
FROM base-image as sub
RUN echo 'bar' > /bar.txt && cat /foo.txt >> /bar.txt
docker-compose.ml
version: "3"
services:
base:
image: base-image
build:
context: .
dockerfile: ./Dockerfile
target: base
deploy:
replicas: 0
sub-with-build:
build:
context: .
dockerfile: ./Dockerfile
target: sub
command: echo 'sub-with-build' && cat /bar.txt
depends_on:
- base
sub-with-image:
image: base-image
command: echo 'sub-with-image' && cat /bar.txt
depends_on:
- base
Error
docker compose up
[+] Running 2/2
✘ sub-with-image Error
The service sub-with-build waits for base-image to be built and then runs as expected but sub-with-image fails. Is this the expected outcome?
@pebo Can confirm this. I had one base image and it was working. Then tried adding a second "stage" (since a few images share part of the features) and suddenly everything fell apart.
Some folks have noted that depends_on controls the order of the build. I am not so sure. Official docs says that depends_on is used to control the startup and shutdown sequence of the containers. So it's a runtime configuration, not a build configuration. See https://docs.docker.com/compose/compose-file/05-services/#depends_on
Apologies if my understanding is wrong.
Some folks have noted that
depends_oncontrols the order of the build. I am not so sure. Official docs says thatdepends_onis used to control the startup and shutdown sequence of the containers. So it's a runtime configuration, not a build configuration. See https://docs.docker.com/compose/compose-file/05-services/#depends_on Apologies if my understanding is wrong.
Even though undocumented, depend_on used to control the build order as well (haven't tested it recently). And unfortunately it's the only way to control the build order without using third-party tools. For some reason, docker-compose maintainers decided to exclude build ordering from the docker-compose feature set, despite most of the code being clearly present and working.
Be careful when using buildkit features. Buildkit uses dockerized build-instances and therefore this instances have some isolation against host system (and host docker). I did not realize at first, what consequences that can have.
If you have two docker-compose services with services that depend on base-image build by another service this can break if you are using separate Dockerfiles!
https://github.com/moby/buildkit/issues/4162#issuecomment-1686304524
export DOCKER_BUILDKIT=0 worked for me on WSL2. Looks like buildkit usage is recent? I would expect it to work as previously though.
