moby
Buildkit build fails when registry mirror is not available (503)
When the registry mirror (configured in docker daemon) is not available Docker builds with buildkit are failing where non buildkit fails are still working.
Expected result is that build is still working like non buildkit builds. This would improve reliability of docker build with buildkit.
This example was build using nginx which always returns 503 on all requests.
nginx (fake registry)
docker nginx: docker run -ti --rm -v "$(pwd)/default.conf:/etc/nginx/conf.d/default.conf" -p 1234:80 nginx
default.conf
server {
location / {
return 503 "";
}
}
docker config
docker/daemon.json:
{
"registry-mirrors": ["http://127.0.0.1:1234"],
"live-restore": true,
"storage-driver": "overlay2",
"log-driver": "json-file",
"log-opts": {
"max-size": "50m",
"max-file": "10"
},
"max-concurrent-downloads": 25,
"max-concurrent-uploads": 25
}
docker build
Dockerfile:
FROM alpine
ENV foo=bar
FROM ubuntu
ENV bar=foo
docker build w/ buildkit
the docker build fails with buildkit:
docker rmi alpine ubuntu; DOCKER_BUILDKIT=1 docker build
result:
Untagged: alpine:latest
Untagged: alpine@sha256:08d6ca16c60fe7490c03d10dc339d9fd8ea67c6466dea8d558526b1330a85930
Untagged: ubuntu:latest
Untagged: ubuntu@sha256:703218c0465075f4425e58fac086e09e1de5c340b12976ab9eb8ad26615c3715
[+] Building 0.1s (3/3) FINISHED
=> [internal] load build definition from Dockerfile 0.1s
=> => transferring dockerfile: 86B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> ERROR [internal] load metadata for docker.io/library/ubuntu:latest 0.0s
------
> [internal] load metadata for docker.io/library/ubuntu:latest:
------
failed to solve with frontend dockerfile.v0: failed to create LLB definition: unexpected status code [manifests latest]: 503 Service Temporarily Unavailable
nginx log:
172.17.0.1 - - [05/Feb/2021:09:59:14 +0000] "HEAD /v2/library/ubuntu/manifests/latest?ns=docker.io HTTP/1.1" 503 0 "-" "containerd/1.4.0+unknown" "-"
docker build w/o buildkit
but not without buildkit:
docker rmi alpine ubuntu; DOCKER_BUILDKIT=0 docker build
result:
Untagged: alpine:latest
Untagged: alpine@sha256:08d6ca16c60fe7490c03d10dc339d9fd8ea67c6466dea8d558526b1330a85930
Untagged: ubuntu:latest
Untagged: ubuntu@sha256:703218c0465075f4425e58fac086e09e1de5c340b12976ab9eb8ad26615c3715
Sending build context to Docker daemon 3.072kB
Step 1/4 : FROM alpine
latest: Pulling from library/alpine
Digest: sha256:08d6ca16c60fe7490c03d10dc339d9fd8ea67c6466dea8d558526b1330a85930
Status: Downloaded newer image for alpine:latest
---> e50c909a8df2
Step 2/4 : ENV foo=bar
---> Using cache
---> 96604f842097
Step 3/4 : FROM ubuntu
latest: Pulling from library/ubuntu
Digest: sha256:703218c0465075f4425e58fac086e09e1de5c340b12976ab9eb8ad26615c3715
Status: Downloaded newer image for ubuntu:latest
---> f63181f19b2f
Step 4/4 : ENV bar=foo
---> Using cache
---> 46a0f7d93e80
Successfully built 46a0f7d93e80
nginx log:
172.17.0.1 - - [05/Feb/2021:09:59:52 +0000] "GET /v2/ HTTP/1.1" 503 197 "-" "docker/20.10.2 go/go1.13.15 git-commit/8891c58 kernel/5.4.0-62-generic os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.2 \x5C(linux\x5C))" "-"
172.17.0.1 - - [05/Feb/2021:09:59:52 +0000] "HEAD /v2/library/alpine/manifests/latest HTTP/1.1" 503 0 "-" "docker/20.10.2 go/go1.13.15 git-commit/8891c58 kernel/5.4.0-62-generic os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.2 \x5C(linux\x5C))" "-"
172.17.0.1 - - [05/Feb/2021:09:59:52 +0000] "GET /v2/library/alpine/manifests/latest HTTP/1.1" 503 197 "-" "docker/20.10.2 go/go1.13.15 git-commit/8891c58 kernel/5.4.0-62-generic os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.2 \x5C(linux\x5C))" "-"
172.17.0.1 - - [05/Feb/2021:09:59:53 +0000] "GET /v2/ HTTP/1.1" 503 197 "-" "docker/20.10.2 go/go1.13.15 git-commit/8891c58 kernel/5.4.0-62-generic os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.2 \x5C(linux\x5C))" "-"
172.17.0.1 - - [05/Feb/2021:09:59:53 +0000] "HEAD /v2/library/ubuntu/manifests/latest HTTP/1.1" 503 0 "-" "docker/20.10.2 go/go1.13.15 git-commit/8891c58 kernel/5.4.0-62-generic os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.2 \x5C(linux\x5C))" "-"
172.17.0.1 - - [05/Feb/2021:09:59:53 +0000] "GET /v2/library/ubuntu/manifests/latest HTTP/1.1" 503 197 "-" "docker/20.10.2 go/go1.13.15 git-commit/8891c58 kernel/5.4.0-62-generic os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.2 \x5C(linux\x5C))" "-"
Client: Docker Engine - Community
Version: 20.10.2
API version: 1.41
Go version: go1.13.15
Git commit: 2291f61
Built: Mon Dec 28 16:17:43 2020
OS/Arch: linux/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.2
API version: 1.41 (minimum version 1.12)
Go version: go1.13.15
Git commit: 8891c58
Built: Mon Dec 28 16:15:19 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.4.3
GitCommit: 269548fa27e0089a8b8278fc4fc781d7f65a939b
runc:
Version: 1.0.0-rc92
GitCommit: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
docker-init:
Version: 0.19.0
GitCommit: de40ad0
happens also with Docker 20.10.3 on Ubuntu 20.04
can confirm: if registry mirror is not responding (timeout/connection refused) build is falling back and continues (with and without buildkit). if registry mirror is responding with 503, build is failing only if buildkit is enabled.
Client: Docker Engine - Community
Version: 20.10.3
API version: 1.41
Go version: go1.13.15
Git commit: 48d30b5
Built: Fri Jan 29 14:33:21 2021
OS/Arch: linux/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.3
API version: 1.41 (minimum version 1.12)
Go version: go1.13.15
Git commit: 46229ca
Built: Fri Jan 29 14:31:32 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.4.3
GitCommit: 269548fa27e0089a8b8278fc4fc781d7f65a939b
runc:
Version: 1.0.0-rc92
GitCommit: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
docker-init:
Version: 0.19.0
GitCommit: de40ad0
Client:
Context: default
Debug Mode: false
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
Server:
Containers: 37
Running: 24
Paused: 0
Stopped: 13
Images: 103
Server Version: 20.10.3
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 5.4.0-62-generic
Operating System: Ubuntu 20.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.56GiB
Name: echelon
ID: COLC:ODLD:WKTW:5MCH:VHBC:GGXV:CKGK:FISV:XFZY:VM4Y:3END:FWDH
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Registry Mirrors:
http://127.0.0.1:1234/
Live Restore Enabled: true
WARNING: No swap limit support
WARNING: No blkio weight support
WARNING: No blkio weight_device support
I have the same
"failed to solve with frontend dockerfile.v0: failed to create LLB definition: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed"
And only disable the Buildkit helps.
export DOCKER_BUILDKIT=0
export COMPOSE_DOCKER_CLI_BUILD=0
I'm using Docker (Preview) for M1 macbook.
Having the same issue here. With multistage build as below
Docker version : 20.10.8
Docker desktop : 3.6.0
OS : Mac Big Sur (11.3)
Dockerfile
FROM nexus.some.company/base AS base
FROM bitnami/minideb:buster
COPY --from=base /opt/container/rootfs /
...
...
ERROR
[+] Building 3.3s (4/4) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 37B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/bitnami/minideb:buster 0.0s
=> ERROR [internal] load metadata for nexus.some.company/base:latest 3.1s
------
> [internal] load metadata for nexus.some.company/base:latest:
------
failed to solve with frontend dockerfile.v0: failed to create LLB definition: nexus.some.company/base:latest: not found
The temporary workaround is to disable buildkit DOCKER_BUILDKIT=0 or pre pull the image docker pull nexus.some.company/base first before running the build command.
For the auth error, if your mirror requires auth it needs to be defined separately for the mirror's hostname. Upstream credentials will not be forwarded.
For the mirror that returns 503, I'm not sure. Could you test it with containerd/ctr (and maybe with buildx container driver) for more datapoints. Afaik the fallback shouldn't require a specific error code.
I am seeing what I believe is the same behavior.
- Version Info
$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.4 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.4 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
$ docker version
Client: Docker Engine - Community
Version: 20.10.13
API version: 1.41
Go version: go1.16.15
Git commit: a224086
Built: Thu Mar 10 14:07:51 2022
OS/Arch: linux/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.13
API version: 1.41 (minimum version 1.12)
Go version: go1.16.15
Git commit: 906f57f
Built: Thu Mar 10 14:05:44 2022
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.5.10
GitCommit: 2a1d4dbdb2a1030dc5b01e96fb110a9d9f150ecc
runc:
Version: 1.0.3
GitCommit: v1.0.3-0-gf46b6ba
docker-init:
Version: 0.19.0
GitCommit: de40ad0
- This is the Docker daemon config (roughly).
$ cat /etc/docker/daemon.json
{
"bip":"172.18.0.1/24",
"debug": true,
"storage-driver": "overlay2",
"registry-mirrors": ["http://docker-registry-mirror.example.com"],
"insecure-registries": ["docker-registry-mirror.example.com"],
"features": { "buildkit": true }
}
- This is a test
Dockerfile
FROM httpd:2.4
CMD ['httpd', '-D', 'FOREGROUND']
- The first time I try to build this, I get an error:
$ docker build .
[+] Building 31.0s (3/3) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 87B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> ERROR [internal] load metadata for docker.io/library/httpd:2.4 31.0s
------
> [internal] load metadata for docker.io/library/httpd:2.4:
------
failed to solve with frontend dockerfile.v0: failed to create LLB definition: unexpected status code [manifests 2.4]: 522
- If I pull it directly, I do see retries on at least one of the layers, but it does eventually download and extract it (within a minute or so)
$ docker pull docker.io/library/httpd:2.4
2.4: Pulling from library/httpd
f7a1c6dad281: Retrying in 5 seconds
f18d7c6e023b: Download complete
bf06bcf4b8a8: Download complete
4566427976c4: Download complete
70a943c2d5bb: Download complete
- It then builds fine.
$ docker build .
[+] Building 0.1s (5/5) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 31B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/httpd:2.4 0.0s
=> [1/1] FROM docker.io/library/httpd:2.4 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:5dd2a64a9ac8645951d6958c7d713fdd9105e58bf225f5008be46a29d7362283 0.0s
- If I start again in a new environment/VM (the base image is not downloaded), edit the daemon config as below, and then restart
dockerd:
$ cat /etc/docker/daemon.json
{
"bip":"172.18.0.1/24",
"debug": true,
"storage-driver": "overlay2",
"features": { "buildkit": true }
}
$ sudo systemctl restart docker
- It will build after the very first attempt
$ docker build .
[+] Building 5.3s (5/5) FINISHED
=> [internal] load build definition from Dockerfile 0.1s
=> => transferring dockerfile: 87B 0.0s
=> [internal] load .dockerignore 0.1s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/httpd:2.4 1.7s
=> [1/1] FROM docker.io/library/httpd:2.4@sha256:b7907df5e39a98a087dec5e191e6624854844bc8d0202307428dd90b38c10140 3.5s
=> => resolve docker.io/library/httpd:2.4@sha256:b7907df5e39a98a087dec5e191e6624854844bc8d0202307428dd90b38c10140 0.0s
=> => sha256:f7a1c6dad28192bd417b78079d6ddc03cbca6d5ea46bba12769b235b6353c00c 31.37MB / 31.37MB 0.9s
=> => sha256:f18d7c6e023b87cd8a0618c93191e5fffaca3bc0749e3172dfd0ff6f3e79e37c 177B / 177B 0.1s
=> => sha256:bf06bcf4b8a8e1aab4125e9d7e66b7c8c7feba9326c000726a39e9e4a8b01874 916.94kB / 916.94kB 0.3s
=> => sha256:b7907df5e39a98a087dec5e191e6624854844bc8d0202307428dd90b38c10140 1.86kB / 1.86kB 0.0s
=> => sha256:05bba44d27fa8eae04f5c293761b83b394b4bde26feb854a0da207a65183d1e3 1.36kB / 1.36kB 0.0s
=> => sha256:faed93b2885914426b66f1a7d09d93dce9e532fa64f4ab269494f4f954a424ef 9.04kB / 9.04kB 0.0s
=> => sha256:4566427976c4f617709f858b5fa79dd374c5d2db937163123ca3fcde6584f991 24.13MB / 24.13MB 1.0s
=> => sha256:70a943c2d5bb3e5cd53d9c1e8876d672e6de34b4b1ba774d4f7f8e18770f5ca4 297B / 297B 0.5s
=> => extracting sha256:f7a1c6dad28192bd417b78079d6ddc03cbca6d5ea46bba12769b235b6353c00c 1.4s
=> => extracting sha256:f18d7c6e023b87cd8a0618c93191e5fffaca3bc0749e3172dfd0ff6f3e79e37c 0.0s
=> => extracting sha256:bf06bcf4b8a8e1aab4125e9d7e66b7c8c7feba9326c000726a39e9e4a8b01874 0.1s
=> => extracting sha256:4566427976c4f617709f858b5fa79dd374c5d2db937163123ca3fcde6584f991 0.7s
=> => extracting sha256:70a943c2d5bb3e5cd53d9c1e8876d672e6de34b4b1ba774d4f7f8e18770f5ca4 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:5dd2a64a9ac8645951d6958c7d713fdd9105e58bf225f5008be46a29d7362283 0.0s
I'd love some help in figuring out how to fix this. Disabling BuildKit is not an option for my usecase.
Can confirm that we're affected by this issue too. We're using a registry mirror, and cannot Disable Buildkit. If the registry mirror has any outages, all docker builds fail.
We use a stopgap measure to fix this this issue, it isn't pretty but until this feature works it is our best option.
docker buildx build \
--cache-from ... \
--tag some-image:some-tag
docker buildx build \
--cache-from ... \
--tag some-image:some-tag \
--cache-to some-options || true
Above is just an example, but the second build should go pretty quick because everything has already been cached on the host itself. The second build will then build again, and push it to the remote registry, if it fails it won't exit the shell script. You can then decide whether to continue with the non-cache image, or exit, retry or leave with a nice message.
In my case, I found that it was because I was using the insecure-registries option but providing an address prefaced with http. With buildkit: false, everything works. With buildkit:true, I got:
failed to solve with frontend dockerfile.v0: failed to create LLB definition: unexpected status code [manifests latest]: 503 Service Unavailable
This registry was an https registry. When I changed the config to preface my insecure registry with https, everything worked again with buildkit: true. So it seems that buildkit may be more strict or not doing some magic that the previous tool was doing.
So maybe in your case your registry is actually only accessible on https and not http. Maybe try it with https?
