Delay/lag when running docker commands

[jjziets]

When I run a docker command there is a 3-5sec delay before it returns an answer.

for example: the image is already cashed so it should just return the result within a fraction of a second.

sudo docker pull nvidia/cuda:11.4.0-base 5-second delay

11.4.0-base: Pulling from nvidia/cuda
Digest: sha256:f0a5937399da5e4efb37fd7b75beb8c484b84dc381243c4b81fc5f9fcad42b66
Status: Image is up to date for nvidia/cuda:11.4.0-base
docker.io/nvidia/cuda:11.4.0-base

This is on an ASUS X99 WS E with a Intel(R) Xeon(R) CPU E5-4667 v3 @ 2.00GHz and NVME. nothing is running on the system. Network speeds are 1Gbps

Docker version 20.10.16, build aa7e414 lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.4 LTS Release: 20.04 Codename: focal

traceroute to 3.211.127.33 (3.211.127.33), 30 hops max, 60 byte packets
 1  _gateway (192.168.2.1)  0.379 ms  0.480 ms  0.602 ms
 2  * * *
 3  port-channel11-100.tercpt-igw2.net.echosp.link (102.67.178.7)  2.871 ms  3.580 ms  3.600 ms
 4  wdcnw-os-cer-1-wan.osnet.co.za (196.25.158.241)  4.012 ms  4.026 ms  4.098 ms
 5  10.189.30.2 (10.189.30.2)  153.109 ms  153.041 ms  153.862 ms
 6  be5956.rcr21.b023101-0.lon13.atlas.cogentco.com (149.11.248.209)  153.793 ms  148.179 ms  152.396 ms
 7  be2348.ccr41.lon13.atlas.cogentco.com (130.117.51.73)  148.195 ms  152.397 ms  148.061 ms
 8  be2099.ccr31.bos01.atlas.cogentco.com (154.54.82.34)  210.327 ms  214.228 ms  215.258 ms
 9  38.140.158.98 (38.140.158.98)  210.866 ms  209.846 ms  209.858 ms

PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=60 time=2.64 m

PING 141.193.213.20 (141.193.213.20) 56(84) bytes of data.
64 bytes from 141.193.213.20: icmp_seq=1 ttl=60 time=2.52 ms

PING google.com (172.217.170.110) 56(84) bytes of data.
64 bytes from jnb02s11-in-f14.1e100.net (172.217.170.110): icmp_seq=1 ttl=118 time=22.6 ms

[binman-docker]

for example: the image is already cashed so it should just return the result within a fraction of a second

This is not the case - if you do a pull on an existing image, it will still issue a GET manifest request to the registry (Hub). This downloads the manifest and compares it to what you currently have. This is because tags are not immutable and are often updated. This ensures you have the latest version of that tag - which you can see when it says "Status: Image is up to date for nvidia/cuda:11.4.0-base"

It takes time to issue the request, go through authentication (even for an anonymous request), download the manifest, and compare the manifest to what currently exists on your machine. If this delay is causing problems and you don't need the latest version of a tag, I would suggest checking whether the image exists on the machine (https://docs.docker.com/engine/reference/commandline/image_ls/) before running the pull command.

You can read more about the pull request flow in the registry API specs here: https://docs.docker.com/registry/spec/api/#pulling-an-image

[jjziets]

Would it help to run a mirror/cache on the local lan?

[binman-docker]

That would mostly cut out the network latency, which seems to be a couple hundred ms for you. Would be easy to test though, just spin up a basic registry container locally and shove a couple images into it, then try pulling from there.