New Docker Hub rate limit & polling issue

[shinebayar-g]

Recently Docker Hub implemented rate limiting policy:

Thus makes Keel's poll policy ineffective. I guess right now keel polls are counting towards this rate limit quota, because it's incrementing download counter of image. Is there any work around to this issue? I noticed that if I already downloaded that image using docker pull command it doesn't increment download counter on second pull.

downloading new image

✗ docker pull busybox
Using default tag: latest
latest: Pulling from library/busybox
Digest: sha256:4f47c01fa91355af2865ac10fef5bf6ec9c7f42ad2321377c21e844427972977
Status: Downloaded newer image for busybox:latest
docker.io/library/busybox:latest

since we've newest image, it doesn't increment download counter on docker hub.

✗ docker pull busybox               
Using default tag: latest
latest: Pulling from library/busybox
Digest: sha256:4f47c01fa91355af2865ac10fef5bf6ec9c7f42ad2321377c21e844427972977
Status: Image is up to date for busybox:latest
docker.io/library/busybox:latest

I guess if keel could able to use the same technique it could solve the rate limit issue. (Only guessing, because counter is not incrementing, doesn't guarantee we will not count towards quota.. Need to verify)

[rusenask]

I think there are two APIs - checksum and tags so yeah, need to check docker's distribution package to see maybe they have another API that doesn't use the limits.

[immanuelfodor]

Huge +1 for opening this issue @shinebayar-g :) I hope we can do something to make polling usable on the long run without a quick ban of the external IP from DockerHub.

[sagikazarmark]

I just want to add that this issue is actually time sensitive: the new rate limit policy will become effective in November.

[shinebayar-g]

Actually It has been in effect over a month now. At least the rate limiting.

[sagikazarmark]

Oh, didn't know that!

BTW found this: https://github.com/crazy-max/diun

Maybe it can provide a solution

[rusenask]

Oh, didn't know that!

BTW found this: https://github.com/crazy-max/diun

Maybe it can provide a solution

looking at the source it seems to be polling too

[immanuelfodor]

Harbor Docker registry updated its docs to address the rate limiting: https://goharbor.io/docs/2.1.0/administration/configure-proxy-cache/

As of Harbor v2.1.1, Harbor proxy cache fires a HEAD request to determine whether any layer of a cached image has been updated in the Docker Hub registry. Using this method to check the target registry will not trigger the Docker Hub rate limiter. If any image layer was updated, the proxy cache will pull the new image, which will count towards the Docker Hub rate limiter.

Maybe this is what Keel should do, too.

[crazy-max]

@rusenask

looking at the source it seems to be polling too

Yes but with HEAD request on manifest (since 4.8.0) which does not count on rate-limit ;)

[kfirfer]

I'm getting this problem too you found something about that?

[kfirfer]

Does this commit fixed the issue ?

[immanuelfodor]

For me, HEADs are fine