pangeo-binder
pangeo-binder copied to clipboard
Failing to build image (Docker Rate Limit for jupyter/repo2docker)
kubectl describe pod -n prod build-pangeo-2ddata-2dpangeo-2ddocker-2dimages-b2b0c6-1c0-0b
Warning Failed 9m28s (x6 over 10m) kubelet Error: ImagePullBackOff
Normal Pulling 9m14s (x4 over 10m) kubelet Pulling image "jupyter/repo2docker:0.11.0-169.gae6c2dc"
Warning Failed 9m13s (x4 over 10m) kubelet Failed to pull image "jupyter/repo2docker:0.11.0-169.gae6c2dc": rpc error: code = Unknown desc = Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
Warning Failed 9m13s (x4 over 10m) kubelet Error: ErrImagePull
Normal BackOff 30s (x45 over 10m) kubelet Back-off pulling image "jupyter/repo2docker:0.11.0-169.gae6c2dc"
According to https://www.docker.com/increase-rate-limits 'anonymous' pulls are limited to 100 per 6 hours.
I'm guessing the limit is tracked by IP address. I tried deleting a node, but the fresh node has the same errors, suggesting any node on the cluster appears as the same IP to DockerHub...
Normal Pulling 89s (x4 over 2m56s) kubelet Pulling image "docker:19.03.5-dind"
Warning Failed 88s (x4 over 2m55s) kubelet Failed to pull image "docker:19.03.5-dind": rpc error: code = Unknown desc = Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
pinging @sgibson91 @consideRatio in case they have run into this issue on other hubs / binder deployments and can suggest any workarounds (other than a paid DockerHub account :)
Hi @scottyhq. We don't run into this on mybinder.org because most of the clusters in the federation push built images to a private container registry, hence the number of images we pull from Docker Hub unauthenticated is very small. We also only pull images per node, zero pulls per launch.
You can pull repo2docker from quay.io which would help here https://quay.io/repository/jupyterhub/repo2docker
We did run into this for a while on OVH which had an admission controller that forces imagePullPolicy: Always
and had to switch that federation member over to a private registry.
But the flagship GKE deployment ('prod' in mybinder.org-deploy) doesn't have any docker credentials and launches 1200 pods/hr. It pulls repo2docker from quay.io and pushes builds to GCR, but that's about it in terms of relevant config, I think.