registry-1.docker.io getting a 404

[nabeelr]

It seems registry-1.docker.io started 404ing sometime today.

It was working earlier today, but I just tried to pull a couple of Docker images, and got 404s.

Please note, I am not using /v2/

Thanks.

[jcarter3]

Are these private images or public? If public, which ones?

[DimaDDM]

Same

[jcarter3]

Which images?

[nabeelr]

They were public images: homeassistant and pihole for me.

[DimaDDM]

Which images?

It's same issue like #2421 and it's really big problem. In my case Docker Hun banned for big IPv6 subnet. It makes the service unavailable on a variety of servers and hosting sites like OVH and Digital ocean even if Ipv6 disabled

[jcarter3]

@DimaDDM That's a 429, not a 404, it's a different issue.

[DimaDDM]

@DimaDDM That's a 429, not a 404, it's a different issue.

It's exactly the same problem actually. Subdomains such as auth, hub, index give 404. At least I've seen this behavior on one of my servers.

[jcarter3]

Docker Hub is not blocking any domains. There is rate limiting that occurs by bucketing IPv6 addresses, but these return 429. If there is a 404, there must be a proxy/routing/DNS issue on the cloud providers.

[nabeelr]

I don't know if this is expected, but I even get a 404 in my web browser too.

[jcarter3]

@nabeelr I did a check on all 404s we have for homeassistant and pihole, and so far all of them are legitimate and due to tags being deleted or being nonexistent. Do you have a specific image/tag that used to work, and I can validate this is the case?

[DimaDDM]

There is rate limiting that occurs by bucketing IPv6 addresses, but these return 429. If there is a 404, there must be a proxy/routing/DNS issue on the cloud providers.

Blocking or rate limits, you can call it whatever you want, the fact is that resources like hub.docker.com and registry-1.docker.io They are currently unavailable to a huge pool of hosting provider addresses due to inept or incorrect blocking of ipv6 and possibly ipv4 subnets. Like @nabeel, I confirm that this problem manifests itself even at the browser level, which prevents docker login or docker logout from being performed. Let me know if there is any way I can help detect or fix this issue.

[jcarter3]

@DimaDDM can you find me on the Docker community slack? We don't do any IPv4 blocking, and the only IPv6 limiting that is done is done in a fairly industry standard way. I believe you that there are problems and will do what I can to help resolve them, but it would be easier in a mechanism with faster feedback.

[MPStudyly]

Chiming in, I don't even get pings through to registry-1.docker.io. Even tried with all v6 addresses I get by doing an nslookup:

~$ nslookup registry-1.docker.io
Server:		127.0.0.53
Address:	127.0.0.53#53

Non-authoritative answer:
Name:	registry-1.docker.io
Address: 44.208.254.194
Name:	registry-1.docker.io
Address: 98.85.153.80
Name:	registry-1.docker.io
Address: 3.94.224.37
Name:	registry-1.docker.io
Address: 2600:1f18:2148:bc01:f43d:e203:cafd:8307
Name:	registry-1.docker.io
Address: 2600:1f18:2148:bc00:5cac:48a0:7f88:7266
Name:	registry-1.docker.io
Address: 2600:1f18:2148:bc02:22:27bd:19a8:870c

~$ ping -c 4 -w 5 2600:1f18:2148:bc00:5cac:48a0:7f88:7266
PING 2600:1f18:2148:bc00:5cac:48a0:7f88:7266(2600:1f18:2148:bc00:5cac:48a0:7f88:7266) 56 data bytes

--- 2600:1f18:2148:bc00:5cac:48a0:7f88:7266 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4094ms

~$ ping -c 4 -w 5 2600:1f18:2148:bc00:5cac:48a0:7f88:7266
PING 2600:1f18:2148:bc00:5cac:48a0:7f88:7266(2600:1f18:2148:bc00:5cac:48a0:7f88:7266) 56 data bytes

--- 2600:1f18:2148:bc00:5cac:48a0:7f88:7266 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4095ms

~$ ping -c 4 -w 5 2600:1f18:2148:bc02:22:27bd:19a8:870c
PING 2600:1f18:2148:bc02:22:27bd:19a8:870c(2600:1f18:2148:bc02:22:27bd:19a8:870c) 56 data bytes

--- 2600:1f18:2148:bc02:22:27bd:19a8:870c ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4077ms

Interestingly enough, manual pulls via "docker pull" seem to work just fine. It's just docker-compose which fails claiming network is unreachable (which it kinda is for these addresses). I can ensure this is not a local issue. Tried pinging the domain via various web ping services, all failed on v6.

[jcarter3]

@MPStudyly are you using a VPS? I know some put in blocks at the provider level until we sorted our IPv6 rate limiting

[MPStudyly]

@MPStudyly are you using a VPS? I know some put in blocks at the provider level until we sorted our IPv6 rate limiting

Nope, this affects all hosts, be it virtual or bare metal. Our IPv6 /48 net is provided by our hoster, though there is no firewall/filtering on their side in place whatsoever.

[jcarter3]

Nvm - we actually disable ping on this endpoint, so this is expected.

[MPStudyly]

Nvm - we actually disable ping on this endpoint, so this is expected.

Damn, then my test was useless :/ any other things I could try to pinpoint this, or is the only alternative explanation the rate limiting issue?