packagist
packagist copied to clipboard
composer
IPv6 issues (and possible solutions)
After long talks with my hosting provider, it seems clear that packagist.org routing IPv6 is at cause. This already caused numerous issues in the past like
- #752
- #743
- #530
- #470
- https://github.com/composer/composer/issues/5889
- https://github.com/composer/composer/issues/7503
- https://github.com/composer/composer/issues/6676
- https://github.com/composer/composer/issues/6519
- https://github.com/composer/composer/issues/6420
- https://github.com/composer/composer/issues/5896
- https://github.com/composer/composer/issues/5825
- https://www.digitalocean.com/community/questions/unable-to-update-composer-connection-timeout-error .... and so many more (just google for it).
Could the owner of packagist.org have a contact with it's provider (OVH). Ticket 72669662 has been filled there about IPv6 to packagist.org. At least in some of the above tickets, it seems that OVH hosting is unable to provide IPv6 connectivity between 2 of its own datacenters.
Ah, so this is why I can't update my dependencies. Any way to force IPv4 usage within composer itself?
Edit: I tossed this into /etc/hosts to get it to work:
aburk@aburk:~$ dig +short repo.packagist.org
142.44.164.255
# echo "142.44.164.255 repo.packagist.org" >> /etc/hosts
Think I have to agree. Packagist/Composer keeps saying the "fix" is to practically disable your box from using ipv6 but a search of everyone having issues pretty much shows it's multiple people, from multiple areas, on multiples of different installs and networks and colo centers, all having problems.
And keeping in mind these folks were probably all using ipv6 before and didn't have issues except for packagist until they came along and found this.
Sounds like the fix isn't as composer suggests (make your box prefer ipv4) but it's time for them to remove their ipv6 hostname records until OVH or someone else can figure out why they have ipv6 issues.
ARIN is out of IPv4 addresses so forcing your network to use it exclusively is not the answer. OVH has been this way since forever and it seems they are not committed to IPv6. https://otacon22.com/2016/02/21/two-hosting-providers-ipv6-setups-compared-ovh-online-net/
I'm not too familiar with how composer pulls code. It seems all packagist does is store json files that point to github/bitbucket or where ever the code is stored. A public S3 bucket would work very well for this. Using CloudFlare or CloudFront may work well too. You wouldn't have to dump your current infrastructure. You would just setup your origin server as the backend for the CDN network.
I'm curious to know how much traffic you guys see in a month.
IPv6 is highly broken in OVH. I've moved all my VPS to somewhere else and all my issues got solved. Just don't use OVH, they don't care about IPv6 in 2018.
btw OVH supports told (in the 72669662 ticket) that the problem was not on their side... (so supposedly coming from the network configuration of the VPS client) @Seldaek ?
Hey folks, I am well aware of the issue, and nagging OVH as well. It's very hard to move forward with this though it seems on their end. I don't know enough about network routing to really say why.
However on our end, options are fairly limited as well. Moving to AWS would cost us quite a bit and CloudFront doesn't let you invalidate things fast enough anyway AFAIK. Disabling IPv6 entirely is not possible either as some people have IPv6-only machines and they would then be unable to reach us.
The best option I see if OVH can't fix this is to migrate to Fastly as they have enough cache control to be workable for us. I'll try to experiment more with that.
In the meantime adding a static IP for repo.packagist.org to your hosts file is probably the best option. Or alternatively disabling IPv6 routing as per https://getcomposer.org/doc/articles/troubleshooting.md#operation-timed-out-ipv6-issues-
+100 Fastly. They have been 100% rock-solid and during the almost five years of experience with them.
OVH claims they fixed a few things in their north american v6 network 4-5 days ago. Does anyone see an improvement? If not please run the following commands and post the outputs here (they might take a few minutes to run):
ping6 repo-ca-bhs-1.packagist.org
mtr -r -P 443 -T -c 100 repo-ca-bhs-1.packagist.org
mtr -r -c 100 repo-ca-bhs-1.packagist.org
If you are not in north america, please let me know if you're having IPv6 issues as well, and show me the output of ping6 repo.packagist.org.
Here are my tests (box checked for successful ping6):
- [x] http://www.traceroute6.net/
- [x] https://www.ultratools.com/tools/traceRoute6Result
- [x] https://centralops.net/co/Traceroute.aspx
- [x] OVH, DE1 && UK1
- [x] Hertzner, DE
- [ ] OVH, BHS3 (vps1) --> unreachable
- [ ] OVH, BHS3 (vps2) --> unreachable
$ mtr -6 -r -P 443 -T -c 100 repo-ca-bhs-1.packagist.org # from BHS3
HOST: Loss% Snt Last Avg Best Wrst StDev
1.|-- [removed] 0.0% 100 0.3 0.3 0.2 3.5 0.2
2.|-- 2607:5300:201::fe 0.0% 100 0.3 0.3 0.2 0.5 0.0
3.|-- 2607:5300:201:c18:ffff:ff 0.0% 100 0.3 0.5 0.3 3.2 0.3
4.|-- 2607:5300:0:1:1:c18:2:1 0.0% 100 0.5 0.5 0.4 0.9 0.0
5.|-- 2607:5300:0:1:1:c8:2:0 0.0% 100 0.5 0.5 0.4 5.2 0.5
6.|-- 2607:5300:201:c8::1f 0.0% 100 0.4 0.4 0.4 0.9 0.0
7.|-- 2607:5300:201::83 0.0% 100 0.6 0.5 0.4 3.5 0.3
8.|-- ??? 100.0 100 0.0 0.0 0.0 0.0 0.0
It happens that among the AAAA records for packagist.org come:
- 2607:5300:201:2100::7:2273
- 2607:5300:201:2100::7:2274
- 2001:41d0:601:2000::536
- 2001:41d0:601:2000::497
The two last ones (Warsaw ?) are reachable from BHS3.
IMHO the link OVH BHS1 <-- ipv6 --> OVH BHS3 is the broken one.
It seems to work Ok from AT&T 2600:1700::/28 AS7018.
I only got 2607:5300:201:2100::7:2273 in the AAAA query for packagist.org. No lost packets from ping -6.
I've been having ipv6 connectivity issues to packagist for the past several months. So far, I haven't noticed trouble to any other hosts. Here's my ping6 / mtr logs, which I think indicate OVH still has a problem.
$ ping6 repo-ca-bhs-1.packagist.org
PING6(56=40+8+8 bytes) 2607:f2c0:f00e:7a00:41b6:a241:6603:4065 --> 2607:5300:201:2100::7:2273
--- repo-ca-bhs-1.packagist.org ping6 statistics ---
100 packets transmitted, 0 packets received, 100.0% packet loss
$ mtr -r -P 443 -T -c 100 repo-ca-bhs-1.packagist.org
Start: 2019-05-01T09:07:53-0400
HOST: acutus-498.local Loss% Snt Last Avg Best Wrst StDev
1.|-- blackbox.lan 0.0% 100 0.4 0.5 0.3 0.8 0.1
2.|-- 2607:f2c0:8006:2::1 0.0% 100 8.4 11.1 8.2 75.1 8.9
3.|-- ae0-2150-bdr01-tor.teksav 0.0% 100 8.4 10.5 8.1 45.2 5.4
4.|-- ??? 100.0 100 0.0 0.0 0.0 0.0 0.0
5.|-- 2607:5300::252f 5.0% 100 8736. 1914. 16.5 8736. 1870.1
6.|-- vl100.bhs-d1-a75.qc.ca 51.0% 100 16.3 16.1 15.8 16.6 0.2
7.|-- be7.bhs-z2g1-a75.qc.ca 44.0% 100 16.1 16.0 15.7 17.0 0.3
8.|-- 2607:5300::1:1:c8:2:0 0.0% 100 16.0 16.0 15.7 17.7 0.3
9.|-- ??? 100.0 100 0.0 0.0 0.0 0.0 0.0
10.|-- ??? 100.0 100 0.0 0.0 0.0 0.0 0.0
11.|-- 2607:5300:201:2100::7:227 47.0% 100 16.5 18.5 15.6 73.6 10.3
$ mtr -r -c 100 repo-ca-bhs-1.packagist.org
Start: 2019-05-01T09:09:42-0400
HOST: acutus-498.local Loss% Snt Last Avg Best Wrst StDev
1.|-- blackbox.lan 0.0% 100 0.3 0.3 0.2 0.6 0.1
2.|-- 2607:f2c0:8006:2::1 0.0% 100 8.3 9.1 8.1 41.6 3.6
3.|-- ae0-2150-bdr01-tor.teksav 0.0% 100 9.8 9.7 8.1 51.4 5.1
4.|-- ??? 100.0 100 0.0 0.0 0.0 0.0 0.0
5.|-- 2607:5300::252f 24.0% 100 17.5 21.4 16.4 98.2 15.4
6.|-- vl100.bhs-d1-a75.qc.ca 0.0% 100 15.9 16.0 15.5 17.3 0.3
7.|-- be5.bhs-z2g1-a75.qc.ca 0.0% 100 15.6 15.9 15.6 17.6 0.2
8.|-- 2607:5300::1:1:c8:1:0 0.0% 100 15.8 15.8 15.6 16.9 0.2
9.|-- ??? 100.0 100 0.0 0.0 0.0 0.0 0.0
Also have the same issue I think it heavily depends from where you are connecting (it sounds like a cdn issue). This is the answer from .nl (and is broken):
dig repo.packagist.org AAAA
; <<>> DiG 9.10.6 <<>> repo.packagist.org AAAA
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 15984
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;repo.packagist.org. IN AAAA
;; ANSWER SECTION:
repo.packagist.org. 120 IN AAAA 2001:41d0:801:1000::895
;; Query time: 43 msec
;; SERVER: 2001:470:7c9e:101::1#53(2001:470:7c9e:101::1)
;; WHEN: Fri May 31 21:07:44 CEST 2019
;; MSG SIZE rcvd: 75
curl -v --connect-timeout 5 [2001:41d0:801:1000::27d]
* Trying 2001:41d0:801:1000::27d...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x7fe809805400)
* Connection timed out after 5002 milliseconds
* Closing connection 0
curl: (28) Connection timed out after 5002 milliseconds
+100 Fastly. They have been 100% rock-solid and during the almost five years of experience with them.
Fastly are used to hosting package repositories, as they already handle PyPI traffic.
