Slow download speeds while pulling images from HUB

[darki73]

Problem description

Good time of the day.

In a past few days I've started noticing that download speed on docker pull became really slow for some reason. I've read that speeds might be throttled due to the amount of images pulled per 6 hours (100 for anonymous and 200 for authenticated).

However, I am experiencing problem that for a couple of days now (I usually pull 3-10 images per day to update containers on my home server) the download speed became really slow.

I've tried multiple things, even authenticated under my account on my server machine, alas it did nothing. One source of a problem might be that I am using a static IP address, however, given the amount of images I am pulling during the day, this should have 0 effect on my download speeds.

As of right now, I am trying to build a container based on nvidia/cuda:11.4.2-cudnn8-runtime-ubuntu20.04 and results are pretty sad to say the least: => => sha256:f68bad238d01924ca5e1d2a3f0bdf2ccf42f742d9a67de0d8d2c0da49661cca0 87.03MB / 1.30GB 329.2s

Is there anything I can do on my side to help to resolve this problem?

docker info output

Docker Info:

Client:
 Context:    default
 Debug Mode: false

Server:
 Containers: 51
  Running: 48
  Paused: 0
  Stopped: 3
 Images: 44
 Server Version: 20.10.5
 Storage Driver: btrfs
  Build Version: Btrfs v4.20.1 
  Library Version: 102
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux nvidia runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.10.28-Unraid
 Operating System: Slackware 14.2 x86_64 (post 14.2 -current)
 OSType: linux
 Architecture: x86_64
 CPUs: 48
 Total Memory: 125.8GiB
 Name: Tower
 ID: ====redacted====
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: darki73
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

Timetamp or time range: Past 3-5 days

Public IP: Static IP address, already sent on in the support through website

Hub Username: darki73

Screenshots of the issue (if applicable)

More Information

Here is also an output from the script from https://github.com/docker/hub-feedback/issues/2161#issuecomment-954890972 https://gist.github.com/darki73/09a0af37b8a7c8c2ea8a3153466293f9

Task List

• [x] This is NOT a security issue
• [x] I do NOT have a Docker subscription
• [x] I have looked through other issues and they do NOT apply to me

[ingshtrom]

I think this is a duplicate of https://github.com/docker/hub-feedback/issues/2161, but the more feedback the better for tracking this down. We can close and continue conversations in the other thread or keep this one, I will be notified either way.

Unfortunately, I messed up that docker_pull.sh script. It was pulling from the wrong place. I have updated it now. Can you please run it again, maybe with ./docker_pull.sh nvidia/cuda 11.4.2-cudnn8-runtime-ubuntu20.04 in order to replay with the exact image you are having issues with.

Some of the data from the gist you provided is useful, so I am investigating that now, but it would definitely help to have another one using the updated docker pull bash script I provided.

Thank you!

[darki73]

@ingshtrom Thank you for a fast reply, I had some issues with the original file, hence the image I pulled. Problem with the new one is that it throws 404 error.

[ingshtrom]

I apologize, try once more 😄 I was able to reproduce the 404. I just updated the gist and ran ./docker_pull.sh nvidia/cuda 11.4.2-cudnn8-runtime-ubuntu20.04; it works now. I was not parsing mono-architecture manifests correctly.

[darki73]

Sorry for long time for a reply, right now I am collecting logs using the provided, updated, script. (still takes a long time to do so) Just a side note, as it might have something to do with the download speed. I am running my home server using Cloudflare DNS as a DNS service provider. I am also using 1.1.1.1 and 1.0.0.1 as my default DNS servers.

Here are the 2 results taken using official speedtest test by Cloudflare, and one by Ookla. As both of them located near DME, this seems to be a valid test. Difference in speeds might be seen on the screenshot.

So, that got me thinking, maybe this is not at all a problem with DockerHub network, but with the Cloudflare.

Here are the logs created by the updated bash script: https://gist.github.com/darki73/96d1202362bf6a63d0a8a74e073d2580

I also looked through the log, and here is what I came up with:

size_download: 28570074           | 28.570074 MB
speed_download: 5899251.000       | 5.899251 MB/s => 47.194008 Mbps

size_download: 7932199            | 7.932199 MB
speed_download: 4112078.000       | 4.112078 MB/s => 32.896624 Mbps

size_download: 11578474           | 11.578474 MB
speed_download: 2562743.000       | 2.562743 MB/s => 20.501944 Mbps

size_download: 185                | 0.000185 MB
speed_download: 174.000           | 0.000174 MB/s => 0.001392 Mbps

size_download: 6432               | 0.006432 MB
speed_download: 4978.000          | 0.004978 MB/s => 0.039824 Mbps

size_download: 1302697637         | 1302.697637 MB
speed_download: 2375785.000       | 2.375785 MB/s => 19.00628 Mbps

size_download: 154                | 0.000154 MB
speed_download: 295.000           | 0.000295 MB/s => 0.00236 Mbps

size_download: 154                | 0.000154 MB
speed_download: 311.000           | 0.000311 MB/s => 0.002488 Mbps

[ingshtrom]

I'm sorry for the delay, I lost this message but rest assured we have continued investigating in #2161 and Cloudflare thinks they have a solution. They would like a confirmation it is the same issue, could you please follow the instructions from here in order to run a test script and do a network packet capture.

Thank you in advance!

[imHarry404]

docker run -d -p 8081:8081 -e ME_CONFIG_MONGODB_ADMINUSERNAME=admin -e ME_CONFIG_MONGODB_ADMINPASSWORD=password --net mongo-network --name mongo-express -e ME_CONFIG_MONGODB_SERVER=mongodb mongo-express
it tool 1hour to download mongo from the hub. very slow