docker
ONBUILD instructions in earlier stages of multi-stage build do not run before COPY --from in later stages
Contributing guidelines
- [X] I've read the contributing guidelines and wholeheartedly agree
I've found a bug and checked that ...
- [X] ... the documentation does not mention anything about my problem
- [X] ... there are no open or closed issues that are related to my problem
Description
Building a docker image fails when the FROM image for the final stage uses a [ONBUILD] COPY --from to copy a directory or file from an earlier stage, if the earlier named stage has an ONBUILD instruction that needs to run before the source of directory or file is created.
If the source file or directory to copy doesn't exist, the error given is some variant of: ERROR: failed to solve: failed to compute cache key: failed to calculate checksum of ref 33a75f97-2e2d-44b6-be3b-0c3fc217ec59::onn3gmwfm28dr2bgekr7461yy: "/build/dist": not found
However, the image build can also appear to succeed but not have the expected final state -- if the directory/file already exists in the stage being copied from it will copy what is there, however an ONBUILD instructions for the earlier stage may modify the file or add files to an empty directory. That updated file/directory does not get copied as would be expected.
Expected behaviour
I would expect that if the earlier stage contains ONBUILD instructions, that those would run before trying to copy contents from the earlier stages into a later stage, to avoid unexpected results.
Actual behaviour
The actual behavior is that the directive for copying a directory/file runs prior to the ONBUILD instructions for earlier stages. This has results that range from an error if the source to copy is created by the ONBUILD instruction, or (worse because its harder to catch) copying outdated file/directory contents that are supposed to be updated by the ONBUILD instructions.
Buildx version
github.com/docker/buildx v0.11.2 9872040
Docker info
Client: Docker Engine - Community
Version: 24.0.5
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.11.2
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.20.2
Path: /usr/libexec/docker/cli-plugins/docker-compose
scan: Docker Scan (Docker Inc.)
Version: v0.23.0
Path: /usr/libexec/docker/cli-plugins/docker-scan
Server:
Containers: 2
Running: 0
Paused: 0
Stopped: 2
Images: 21
Server Version: 24.0.5
Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 8165feabfdfe38c65b599c4993d227328c231fca
runc version: v1.1.8-0-g82f18fe
init version: de40ad0
Security Options:
seccomp
Profile: builtin
Kernel Version: 4.18.0-513.11.1.el8_9.x86_64
Operating System: Red Hat Enterprise Linux 8.9 (Ootpa)
OSType: linux
Architecture: x86_64
CPUs: 20
Total Memory: 62.32GiB
Name: <redacted>
ID: 33a75f97-2e2d-44b6-be3b-0c3fc217ec59
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Builders list
NAME/NODE DRIVER/ENDPOINT STATUS BUILDKIT PLATFORMS
default * docker
default default running v0.11.6+0a15675913b7 linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/amd64/v4, linux/386
Configuration
my-builder Dockerfile:
FROM python:3.11.7-slim
ONBUILD RUN mkdir -p build/dist/
ONBUILD RUN touch build/dist/a_file
Build with: sudo docker build -t my-builder:latest .
my-base Dockerfile:
FROM python:3.11.7-slim
ONBUILD COPY --from=builder /build/dist/ /root/dist
Build with: sudo docker build -t my-base:latest .
Multi-stage build Dockerfile:
FROM my-builder:latest as builder
FROM my-base:latest
Build with: sudo docker build --no-cache .
This should result in an error along the lines of "failed to computer cache key". However, if in the my-builder Dockerfile the ONBUILD prefix is removed from the line that creates that directory (but leave it on the 2nd RUN instruction!), then we can see that building this multi-stage Docker image appears to succeed. However, when we go to look at the contents of the image, /root/dist does not contain the a_file that gets created as part of the ONBUILD instruction in the earlier builder stage.
Build logs
[+] Building 0.1s (6/6) FINISHED docker:default
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 137B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/my-base:latest 0.0s
=> [builder 1/1] FROM docker.io/library/my-builder:latest 0.0s
=> CACHED [stage-1 1/1] FROM docker.io/library/my-base:latest 0.0s
=> ERROR [stage-1 2/1] COPY --from=builder /build/dist/ /root/dist 0.0s
------
> [stage-1 2/1] COPY --from=builder /build/dist/ /root/dist:
------
Dockerfile:1
--------------------
1 | >>> FROM my-builder:latest as builder
2 |
3 | FROM my-base:latest
--------------------
ERROR: failed to solve: failed to compute cache key: failed to calculate checksum of ref 33a75f97-2e2d-44b6-be3b-0c3fc217ec59::a13omyrr4f5v0dbdmebku3tt6: failed to walk /var/lib/docker/tmp/buildkit-mount3383600270/build: lstat /var/lib/docker/tmp/buildkit-mount3383600270/build: no such file or directory
Additional info
No response
~Even just regular COPY without the ONBUILD fails to work — it either copies a stale file/folder from the earlier stage containing ONBUILD instructions, or it fails if the file/folder doesn’t exist before the ONBUILD instructions in the earlier stage are run.~
Revisiting this after some rest, it actually looks like inserting a regular COPY (not ONBUILD) in the final stage after the FROM command does work, and the ONBUILD instructions from the earlier stage get evaluated first. From the docs (https://docs.docker.com/reference/dockerfile/#onbuild), The trigger will be executed in the context of the downstream build, as if it had been inserted immediately after the FROM instruction in the downstream Dockerfile implies that ONBUILD COPY should have identical behavior.