docker
`docker build` fails to authenticate with private repository
Description
I have noticed an issue using private repositories with authentication since upgrading to the latest docker version with Docker Engine v23.
Using basic auth (no credential helper) docker pull can pull the image from the private repository (Nexus in my case) after running docker login. But running docker build . fails with the following error:
] Building 0.1s (3/3) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 100B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> ERROR [internal] load metadata for ***/moembed/runtime:4.0.0-arm64v8 0.1s
------
> [internal] load metadata for ***/moembed/runtime:4.0.0-arm64v8:
------
Dockerfile:1
--------------------
1 | >>> FROM ***/moembed/runtime:4.0.0-arm64v8
2 | CMD echo
3 |
--------------------
ERROR: failed to solve: ***/moembed/runtime:4.0.0-arm64v8: pulling from host *** failed with status code [manifests 4.0.0-arm64v8]: 401 Unauthorized
Since docker pull will pull the image from the private registry after docker login the following steps will lead to a successful build:
docker login some.registry.comdocker pull some.registry.com/imagedocker build .
I found that the behavior is similar to docker/cli#3716 but only started for me after installing docker with Docker Engine v23.
Reproduce
docker login some.registry.com- Create
Dockerfilewith relevant content (withFROMclause referring to a private registry image) docker build .
Expected behavior
docker build should use the given credentials after a docker login and successfully pull the referenced image from the given registry.
docker version
Client: Docker Engine - Community
Version: 23.0.0
API version: 1.42
Go version: go1.19.5
Git commit: e92dd87
Built: Wed Feb 1 17:46:28 2023
OS/Arch: linux/arm64
Context: default
Server: Docker Engine - Community
Engine:
Version: 23.0.0
API version: 1.42 (minimum version 1.12)
Go version: go1.19.5
Git commit: d7573ab
Built: Wed Feb 1 17:46:28 2023
OS/Arch: linux/arm64
Experimental: false
containerd:
Version: 1.6.16
GitCommit: 31aa4358a36870b21a992d3ad2bef29e1d693bec
runc:
Version: 1.1.4
GitCommit: v1.1.4-0-g5fd4c4d
docker-init:
Version: 0.19.0
GitCommit: de40ad0
docker info
Client:
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.10.2
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.15.1
Path: /usr/libexec/docker/cli-plugins/docker-compose
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 3
Server Version: 23.0.0
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 31aa4358a36870b21a992d3ad2bef29e1d693bec
runc version: v1.1.4-0-g5fd4c4d
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 5.15.0-58-generic
Operating System: Ubuntu 22.04.1 LTS
OSType: linux
Architecture: aarch64
CPUs: 3
Total Memory: 5.779GiB
Name: ubuntu-vm
ID: e8b9743c-f49f-407c-8d8f-e0c56ad58277
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Additional Info
No response
@crazy-max @jedevc PTAL; at a glance I don't see [auth] in the output (ISTR BuildKit includes that if it's handling authentication?). Should this be moved to the buildx issue tracker?
@alexhendel out of curiosity, if you run with docker buildx build instead of just docker build (after making sure the image hasn't been pulled locally) does it still break?
Actually shows the same result. buildx build will error out. But running docker pull will pull the image without errors and buildx build directly after that doesn't fail..
~/source/docker-test$ sudo docker buildx build .
[+] Building 0.1s (3/3) FINISHED
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 100B 0.0s
=> ERROR [internal] load metadata for ***/moembed/runtime:4.0.0-arm64v8 0.1s
------
> [internal] load metadata for ***/moembed/runtime:4.0.0-arm64v8:
------
Dockerfile:1
--------------------
1 | >>> FROM ***/moembed/runtime:4.0.0-arm64v8
2 | CMD echo
3 |
--------------------
ERROR: failed to solve: ***/moembed/runtime:4.0.0-arm64v8: pulling from host *** failed with status code [manifests 4.0.0-arm64v8]: 401 Unauthorized
~/source/docker-test$ sudo docker pull ***/moembed/runtime:4.0.0-arm64v8
4.0.0-arm64v8: Pulling from moembed/runtime
Digest: sha256:2fac5f0e2e94fbd4d03658b7f8aa840a7c6089bfe0ad4ceffd055ea0d649ffe2
Status: Downloaded newer image for ***/moembed/runtime:4.0.0-arm64v8
***/moembed/runtime:4.0.0-arm64v8
~/source/docker-test$ sudo docker buildx build .
[+] Building 0.0s (5/5) FINISHED
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 100B 0.0s
=> [internal] load metadata for ***/moembed/runtime:4.0.0-arm64v8 0.0s
=> CACHED [1/1] FROM ***/moembed/runtime:4.0.0-arm64v8 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:a48d05fdcf8e508d0a8114f132f5ab2e4790e28ed425638a2ac04a0ee212b92
@alexhendel what does your ~/.docker/config.json look like?
⚠️ ⚠️ the file may contain your credentials if you're not using a credentials helper, so be sure to sanitize those parts before posting
The config json below (I replaced the actual values with ***):
The registry value is what I would expect and the auth value appears to be hashed string.
{
"auths": {
"***": {
"auth": "***"
}
}
}
Thanks! Yes, that looks fairly standard (was curious if perhaps a credential-helper was in use somewhere).
Do you happen to know what kind of registry is running on your some.registry.com ? (of course, if you can disclose, just looking for information that could help investigate)
Are you using a remote server and is it possible the registry does additional authentication based on IP? If yes, does setting BUILDKIT_NO_CLIENT_TOKEN=true make any difference?
"***": {
Is this a clean domain like in the reference you are pulling, or URL with some protocol/path?
Are you using a remote server and is it possible the registry does additional authentication based on IP? If yes, does setting
BUILDKIT_NO_CLIENT_TOKEN=truemake any difference?
We are using Sonatype Nexus to host a private registry. I am not sure about the IP-based authentication.
Where would I configure BUILDKIT_NO_CLIENT_TOKEN=true?
"***": {
Is this a clean domain like in the reference you are pulling, or URL with some protocol/path?
This value is a clean domain. No path or protocol.
We did some additional testing which might give some more details. We did try podman and ran:
podman pull
podman build .
This behaves like the previous Docker version, build and pull work fine. Checking the network traffic the only difference we could find was an additional HEAD request with v23 (which was not sent previously and is also not sent by podman).
We have experienced the exactly same issue with docker 23.0.1 against private images in GitHubs ghcr.io-registry. Adding BUILDKIT_NO_CLIENT_TOKEN=true fixed the issue for us. Also downgrading to 20.x.x solved the issue.
I can confirm that BUILDKIT_NO_CLIENT_TOKEN=1 works for us (we are using docker 23, and the private registry is artifactory with client certificate required for authentication).
I can confirm that
BUILDKIT_NO_CLIENT_TOKEN=1works for us (we are using docker 23, and the private registry is artifactory with client certificate required for authentication).
@tgquan67 What version of Artifactory are you using? I can't seem to get it to work with Artifactory 7.49.3.
BUILDKIT_NO_CLIENT_TOKEN=1 suggests that the difference is in some network layer and only the daemon can connect to the auth server. The code itself should be identical.
So BUILDKIT_NO_CLIENT_TOKEN=1 is supposed to be an environment variable for the daemon process? Not the client?
No, on the client.
BUILDKIT_NO_CLIENT_TOKEN=1 docker buildx build ...
Yup, that's what I was trying but still no luck. What's odd for me is it does seem like my layers are pushed to my Artifactory successfully, but when it tries to push the manifest then Artifactory responds with a 403. And since I'm on a nearly 6 month old version of Artifactory, I'm not 100% certain the issue with is with buildkit/buidx.
I'll have to update my Artifactory and try again.
@tonistiigi related to these?
- https://github.com/containerd/containerd/issues/4982
- https://github.com/containerd/containerd/pull/3913
- https://github.com/containerd/containerd/pull/3912
- https://github.com/containerd/containerd/pull/4983
Last one has further details (it was rejected at the time); https://github.com/containerd/containerd/pull/4983#issuecomment-798727645
What version of Artifactory are you using? I can't seem to get it to work with Artifactory
7.49.3.
I'm on artifactory 7.41 and it worked. Btw, in my case I only test if it can pull the base image from artifactory, I did not try to push it with buildx.
Maybe disable buildkit with DOCKER_BUILDKIT=0 and see if it works first, and then check how you authenticate with artifactory.
I updated my Artifactory to their newest version, but still no luck. However, I did try docker login with the admin account of my Artifactory and that worked. So I don't think my issue is with buildx but something with my Artifactory.
Yup, it was just a permission issue on my Artifactory that caused the 403 when buildkit was trying to HEAD to see if the manifest was already there before the PUT to upload it. I fixed the permission issue and it worked. It even works for me without setting BUILDKIT_NO_CLIENT_TOKEN=1.
Now I test it again on docker 24, it works even without BUILDKIT_NO_CLIENT_TOKEN=1 on the default builder; but if i create a new builder (for example with docker buildx create --name test --driver=docker-container), I have no way to make it work when this new builder is in use (with docker buildx use test).
I'm still seeing this on docker 25.0.3 and buildx v0.12.1
$ docker --version
Docker version 25.0.3, build 4debf41
$ docker buildx version
github.com/docker/buildx v0.12.1-desktop.4 6996841df2f61988c2794d84d33205368f96c317
@alexhendel just curious if you ever resolved this on your side? We're using Nexus and facing the same thing. Works with legacy builder, just not BuildKit.
Another update: at some point after docker 25 update, it started working in my case (artifactory 7.41) with BUILDKIT_NO_CLIENT_TOKEN=1. Also since I have client certificate requirement for my registry, I also have this file at /etc/buildkitd.toml:
debug = true
[registry."artifactory.myregistry.net"]
ca=["/opt/myregistry/certs/ca.crt"]
[[registry."artifactory.myregistry.net".keypair]]
key="/opt/myregistry/certs/client.key"
cert="/opt/myregistry/certs/client.pem"
Create a new builder with
docker buildx create --use --bootstrap --name basic-builder --driver docker-container --config /etc/buildkitd.toml
And I also export these 3 variables:
export BUILDKIT_NO_CLIENT_TOKEN=1
export DOCKER_BUILDKIT=1
export BUILDX_BUILDER=basic-builder
After this in dockerfiles I can pull from artifactory.myregistry.net using credentials I logged in with docker login artifactory.myregistry.net.