docker build does not respect proxy settings (image pull failure) but docker pull works fine

[bswilliamson]

• [x] I have tried with the latest version of Docker Desktop
• [x] I have tried disabling enabled experimental features
• [x] I have uploaded Diagnostics
• Diagnostics ID: 7B0E70A2-2A7A-48B3-9346-AF07EFC553FF/20210411193149

Actual behavior

docker build does not respect Docker Desktop proxy settings and fails to pull an image but docker pull can successfully pull the same image.

Expected behavior

docker build should be able to pull any images that docker pull can.

Information

• Is it reproducible: yes
• Is the problem new: this is my first time using docker on Windows so I don't know
• Did the problem appear with an update: n/a
• Windows Version: Windows 10 20H2 build 19042.867
• Docker Desktop Version: 3.3.0.62916
• Backend: WSL2
• Host: Win 10 developer VM running on Hyper-V

My network environment requires me to use a proxy to access external sites. I have a proxy configured in Docker Desktop and system wide in Windows settings using a PAC file. I can successfully pull images using docker pull but I cannot build images because docker build fails with a timeout on auth.docker.io. Restarting the docker daemon has no impact and I have double checked my proxy config as shown below.

Proxy configuration check

PS C:\Users\User\src\docker> [System.Net.WebRequest]::DefaultWebProxy.GetProxy("https://docker.io").AbsoluteUri
http://redacted:8080/
PS C:\Users\User\src\docker> Get-Content $env:AppData\Docker\settings.json | ConvertFrom-JSON | Select-Object -Property @('proxyHttpMode', 'overrideProxyHttp', 'overrideProxyHttps', 'overrideProxyExclude')

proxyHttpMode overrideProxyHttp            overrideProxyHttps           overrideProxyExclude
------------- -----------------            ------------------           --------------------
         True http://redacted:8080 http://redacted:8080 localhost,127.0.0.1

PS C:\Users\User\src\docker> (Invoke-WebRequest https://docker.io).StatusCode
200

Steps to reproduce the behavior

Pull image successfully

PS C:\Users\User\src\docker> docker pull debian:buster
buster: Pulling from library/debian
bd8f6a7501cc: Pull complete
Digest: sha256:ba4a437377a0c450ac9bb634c3754a17b1f814ce6fa3157c0dc9eef431b29d1f
Status: Downloaded newer image for debian:buster
docker.io/library/debian:buster

Build image based on the same image after removing (fails)

PS C:\Users\User\src\docker> docker rmi debian:buster
Untagged: debian:buster
Untagged: debian@sha256:ba4a437377a0c450ac9bb634c3754a17b1f814ce6fa3157c0dc9eef431b29d1f
Deleted: sha256:0d587dfbc4f4800bfe9ab08662e8396ffc37060c493f8ef24b2823fef3320df6
Deleted: sha256:e2c6ff46235709f5178ab7c1939f4fba7237ffde84e13b1582fa5f0837c1d978
PS C:\Users\User\src\docker> Get-Content Dockerfile
FROM debian:buster
CMD ["sh", "-c", "echo test"]
PS C:\Users\User\src\docker> docker build . -t test
[+] Building 31.4s (4/4) FINISHED
 => [internal] load build definition from Dockerfile                                                               0.1s
 => => transferring dockerfile: 87B                                                                                0.0s
 => [internal] load .dockerignore                                                                                  0.1s
 => => transferring context: 2B                                                                                    0.0s
 => ERROR [internal] load metadata for docker.io/library/debian:buster                                            31.2s
 => [auth] library/debian:pull token for registry-1.docker.io                                                      0.0s
------
 > [internal] load metadata for docker.io/library/debian:buster:
------
failed to solve with frontend dockerfile.v0: failed to create LLB definition: failed to authorize: rpc error: code = Unknown desc = failed to fetch oauth token: Get https://auth.docker.io/token?scope=repository%3Alibrary%2Fdebian%3Apull&service=registry.docker.io: dial tcp 54.85.56.253:443: i/o timeout

[thaJeztah]

Do you see the same problem if you try to do the build with the "classic" builder?

You can temporarily disable BuildKit by setting the DOCKER_BUILDKIT=0 environment variable in the shell where you perform the build.

[bswilliamson]

No that works fine. At least I can use that as a workaround.

[thaJeztah]

@tonistiigi PTAL ^^

[tonistiigi]

Buildkit does not send your password to the remote daemon, instead, it requests a temporary token from the client-side. That means that if you can't access the internet (auth server) your proxy settings need to be set on the docker binary you are invoking.

[bswilliamson]

The system proxy is already set in the terminal environment (see proxy configuration check above) and Invoke-WebRequest can successfully make requests. So docker must not be detecting the system proxy.

[bswilliamson]

I've done some more testing and if I override the system proxy set via Windows settings with the http_proxy and https_proxy env vars in powershell then it does indeed work. However this means that I currently have proxy settings in three separate places in order for Docker Desktop to work:

• GUI = Windows system proxy
• daemon = GUI > Settings > Resources > Proxies
• Windows CLI = http_proxy/https_proxy

Is this disparity between the GUI and the CLI the intended behaviour?

[stephen-turner]

No, it's not intended, we've obviously missed passing the configuration around in the change from classic builder to buildkit. We'll add another ticket to improve it. Thanks for the report.

[docker-robott]

Issues go stale after 90 days of inactivity. Mark the issue as fresh with /remove-lifecycle stale comment. Stale issues will be closed after an additional 30 days of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows. /lifecycle stale

[thaJeztah]

/lifecycle frozen

[paulfelton]

@stephen-turner Did you add another ticket, and if so, could you share the reference please?

[stephen-turner]

We have an internal ticket, not publicly visible.

[thaJeztah]

/cc @crazy-max

[paulfelton]

Ta. Any idea when it might progress, just to set our expectations?

[stephen-turner]

Sorry, no specific date at the moment.

[cailloumajor]

I encounter the same (annoying) problem. However, everything worked fine at the time I started to use BuildKit when it was an experimental feature.

[KnoerleMan]

Any news on this? Took me around 3 hours until google finally showed the right entry about this issue.

[connors2015]

Just ran into this tonight as well. Works perfectly fine with 'run' command then can't build because it can't pull dependencies.

Does not happen on Linux for me, only windows.

[dnoliver]

I was aware of this issue for a while now. But it became too annoying with Docker DevEnvironments. The devenv feature is kind-of GUI only (I had a bunch of other problems when trying to use the docker dev cli, so GUI usually works fine). The problem is that it fails to auth you when creating the environment, so you have to delete the devenv, move out of the VPN, and then re-create it, and then it starts building fine. But if you have a build step that requires you to be inside the VPN (e.g. fetching a binary from an internal artifact storage), then I guess you will be out of luck, and DevEnv will became unusable

[bunglegrind]

I have the same issue with docker compose build. Solved by setting DOCKER_BUILDKIT=0, in order to download the container layers, and then again setting DOCKER_BUILDKIT=1 for the remaining commands.

[Ali-Flt]

just run

export http_proxy=...
export https_proxy=...

before running the docker build command.

[dnoliver]

@Ali-Flt my comment was for docker dev environments, which is another place where this problem shows up. Based from the docs at https://docs.docker.com/engine/reference/commandline/cli/#environment-variables, the variables needs to be capitalized, so it will be:

export HTTP_PROXY=XXX
export HTTPS_PROXY=XXX
export NO_PROXY=XXX
docker dev create $(pwd) --open

So this command fails when you don't have the base image in your local registry already. A workaround is: docker pull <base image>, then docker compose -f compose-dev.yml build <dev environment> and then docker dev create ...

At the time I wrote that comment, the docker dev cli was not working properly in my environment, so the only way to do this was to use the GUI. As today the docker dev cli works fine, it just still run into the problem described in the issue. So all our developers have to pull the base image before creating the dev environment (or switch out of the proxied VPN while building the environment)

[jokerwyt]

@Ali-Flt my comment was for docker dev environments, which is another place where this problem shows up. Based from the docs at https://docs.docker.com/engine/reference/commandline/cli/#environment-variables, the variables needs to be capitalized, so it will be:

export HTTP_PROXY=XXX
export HTTPS_PROXY=XXX
export NO_PROXY=XXX
docker dev create $(pwd) --open

So this command fails when you don't have the base image in your local registry already. A workaround is: docker pull <base image>, then docker compose -f compose-dev.yml build <dev environment> and then docker dev create ...

At the time I wrote that comment, the docker dev cli was not working properly in my environment, so the only way to do this was to use the GUI. As today the docker dev cli works fine, it just still run into the problem described in the issue. So all our developers have to pull the base image before creating the dev environment (or switch out of the proxied VPN while building the environment)

Capitalized env still cannot work for docker pull. Is there any update now?

[mahdic200]

I'm in Iran and I have this issue two . I tried docker build on my ubuntu server outside of Iran and its working . but in Iran and even with the proxy settings (both terminal ENV vars and conf files) not working for me . though when I put the https://auth.docker.io/token link manually in the browser (I have set proxy settings for my desktop and link works fine with browser) I will receive the response and I see the token value . what is this exactly ?

certainly there is a big bug . it does not make any sense at all . this issue is open since 3 GOD DAMN years ago and nobody has a good answer for it .

I just love that answer from this dude , like we were incapable to set a HTTP/HTTPS proxy for ourselves @ali

just run

export http_proxy=...
export https_proxy=...

before running the docker build command.

[dhpagani]

Hello, i'm try to put my two cents in this problem...

My environment has two enterprise proxys: proxy1.test.com and proxy2.test.com.

• proxy1.test.com: default proxy for default use
• proxy2.test.com: cache for docker blobs only and provide authentication for a specific server (server1.com), just set in docker daemon configuration service(systemd). If it fails, it will use proxy1.test.com transparently.

So, in the linux VMs has this configuration:

/etc/environment HTTPS_PROXY=http://proxy1.test.com

/etc/systemd/system/docker.service.d/http-proxy.conf [Service] Environment="HTTP_PROXY=proxy2.test.com" Environment="HTTPS_PROXY=proxy2.test.com"

So, the default proxy should be proxy1.test.com and docker requersts should use proxy2.test.com. The test scenario is: run docker build using the dockerfile FROM server1.com/blabla. The expected result is download the image server.com/blabla and create a new image locally.

With buildkit: The result is a problem with authentication, becase it is using the CLI HTTPS_PROXY(proxy1.test.com).

Without buildkit(set DOCKER_BUILDKIT=0): The result is the same as expected. It downloads and creates the image bevause use the proxy2.test.com

So basically, buildkit uses the proxy variables from CLI to authenticate to registries, and then to download the images, docker uses the daemon setting.

To solve the problem, I set my default proxy to be proxy2.test.com and hope it can handle all the requests

[mahdic200]

actually I searched a lot and I found a mirror for docker in my country , and also I used docker desktop in windows and everything worked perfect . but for some reason which I don't know , in Linux nothing works fine with docker . and actually it doesn't make any sense . I gave up using docker and instead I installed prerequisites on my remote ubuntu server and configured my project manually . it's working fine now .

but if you want to use docker at any cost , I suggest you to use docker desktop and set these settings in docker desktop application on windows .