changelog.com
changelog.com copied to clipboard
thechangelog
IPv6 support
The changelog.com website seems to be inaccessible over IPv6. Since the site seems to be hosted on Linode, which has great IPv6 support, it should hopefully be a fairly simple fix.
For cdn.changelog.com it should be a matter of asking Fastly support to enable IPv6 on the map, per their IPv6 documentation.
TL;DR I didn't forget about this, still planning on seeing it through.
Since we will have a Linode node balancer in front of the app host, it makes sense to do this part of that story, rather than solving IPv6 on the app host, then doing it again when we set up the node balancer.
I'm actually able to fetch changelog.com over IPv6 with some trickery, e.g. curl https://li1926-248.members.linode.com -H "Host:changelog.com" -k -6. So even currently it's only a matter of adding the IPv6 address from li1926-248.members.linode.com (2600:3c03::f03c:91ff:fe2b:6d2c) as an AAAA record.
For the CDN, it's still a matter of asking Fastly support to enable IPv6 on changelog.map.fastly.net.
Spot on: there are a few small steps to see the IPv6 support through properly.
As a show of appreciation for your comments:
dig changelog.com AAAA +short
2600:3c03::f03c:91ff:fe2b:6d2c
💚
Awesome!
I did quite a bit of poking around and here are all other IPv6-related issues I could find:
~~ci.changelog.com – add 2600:3c03::f03c:91ff:fee0:8640 for IPv6~~
- [x]
netdata.changelog.com,changelog.fm,gotime.fm,jsparty.fm,rfc.fm– add2600:3c03::f03c:91ff:fe2b:6d2cfor IPv6 - [ ]
thechangelog.comandnightly.changelog.comlack IPv6. Since those seem to be on DigitalOcean, it'll be necessary to follow their IPv6 enablement guide. - [ ]
changelog-assets.s3.amazonaws.comis used for static resources in blog posts (and emails?). Should be replaced withchangelog-assets.s3.dualstack.us-east-1.amazonaws.comfor IPv6 support.
I've also made a PR to enable HTTPS and HSTS preloading.
I checked netdata, and it seems after enabling IPv6, an IPv4 socket leak has manifested:
The leaks seems to be because of containers:
This doesn't seem to have impacted the website itself at all though.
That was a good catch! This seems to be related to TCP6 sockets remaining stuck in FIN_WAIT2, as described in docker/for-linux#335. This is what happens when Docker gets restarted:
What happens after 30' of Docker running:
ss -st | grep -v ESTAB
Total: 524 (kernel 4662)
TCP: 177 (estab 15, closed 119, orphaned 2, synrecv 0, timewait 26/0), ports 0
Transport Total IP IPv6
* 4662 - -
RAW 1 0 1
UDP 20 8 12
TCP 58 32 26
INET 79 40 39
FRAG 0 0 0
State Recv-Q Send-Q Local Address:Port Peer Address:Port
CLOSE-WAIT 0 0 172.17.0.1:59988 172.17.0.2:https
CLOSE-WAIT 0 0 172.17.0.1:32808 172.17.0.2:https
CLOSE-WAIT 0 0 172.17.0.1:36280 172.17.0.2:https
CLOSE-WAIT 0 0 172.17.0.1:55510 172.17.0.2:https
CLOSE-WAIT 0 0 172.17.0.1:33508 172.17.0.2:https
CLOSE-WAIT 0 0 172.17.0.1:37588 172.17.0.2:https
CLOSE-WAIT 0 0 172.17.0.1:36324 172.17.0.2:https
CLOSE-WAIT 0 0 172.17.0.1:36100 172.17.0.2:https
CLOSE-WAIT 0 0 172.17.0.1:37310 172.17.0.2:https
CLOSE-WAIT 0 0 172.17.0.1:36870 172.17.0.2:https
CLOSE-WAIT 0 0 172.17.0.1:54910 172.17.0.2:https
CLOSE-WAIT 0 0 172.17.0.1:53328 172.17.0.2:https
CLOSE-WAIT 0 0 172.17.0.1:54898 172.17.0.2:https
CLOSE-WAIT 30 0 172.104.216.248:37352 169.46.82.177:22910
CLOSE-WAIT 0 0 172.17.0.1:36480 172.17.0.2:https
FIN-WAIT-1 0 1001 [2600:3c03::f03c:91ff:fe2b:6d2c]:https [2a01:4b00:840f:a200:74e7:c2c4:7e49:66d9]:52904
FIN-WAIT-2 0 0 [2600:3c03::f03c:91ff:fe2b:6d2c]:https [2605:8d80:6a3:12fa:b41a:1e1f:8206:2e6b]:38628
FIN-WAIT-2 0 0 [2600:3c03::f03c:91ff:fe2b:6d2c]:https [2600:1702:c20:4000:99e4:eab4:3ac3:20b]:41564
FIN-WAIT-2 0 0 [2600:3c03::f03c:91ff:fe2b:6d2c]:https [2600:387:5:80d::78]:51889
FIN-WAIT-2 0 0 [2600:3c03::f03c:91ff:fe2b:6d2c]:https [2601:184:4780:5002:ad15:cfc4:806f:2080]:37073
FIN-WAIT-2 0 0 [2600:3c03::f03c:91ff:fe2b:6d2c]:https [2001:1c00:1f09:d500:ec05:daea:ad3:1821]:58338
FIN-WAIT-2 0 0 [2600:3c03::f03c:91ff:fe2b:6d2c]:https [2600:6c64:7b7f:fa70:bc93:ad8f:c665:293]:47578
FIN-WAIT-2 0 0 [2600:3c03::f03c:91ff:fe2b:6d2c]:https [2601:18a:c100:7360:54d0:d99e:2cbf:684]:38931
FIN-WAIT-2 0 0 [2600:3c03::f03c:91ff:fe2b:6d2c]:https [2001:1c00:1f09:d500:ec05:daea:ad3:1821]:58206
FIN-WAIT-2 0 0 [2600:3c03::f03c:91ff:fe2b:6d2c]:https [2600:1:951b:137d:850b:5f7f:e8ed:21c9]:44393
FIN-WAIT-2 0 0 [2600:3c03::f03c:91ff:fe2b:6d2c]:https [2601:282:201:f0fb:ad90:56a3:2c47:7ecb]:45287
FIN-WAIT-2 0 0 [2600:3c03::f03c:91ff:fe2b:6d2c]:https [2001:470:1f1b:321:5da3:b731:f1d4:b3f2]:37391
FIN-WAIT-2 0 0 [2600:3c03::f03c:91ff:fe2b:6d2c]:https [2a01:598:9985:ea8b:bc52:8d92:6316:c50a]:42811
FIN-WAIT-2 0 0 [2600:3c03::f03c:91ff:fe2b:6d2c]:https [2601:282:201:f0fb:ad90:56a3:2c47:7ecb]:45315
FIN-WAIT-2 0 0 [2600:3c03::f03c:91ff:fe2b:6d2c]:https [2606:a000:4f02:1ff0:3836:cbfc:1c31:8fc2]:35790
FIN-WAIT-2 0 0 [2600:3c03::f03c:91ff:fe2b:6d2c]:https [2601:282:201:f0fb:ad90:56a3:2c47:7ecb]:45291
FIN-WAIT-2 0 0 [2600:3c03::f03c:91ff:fe2b:6d2c]:https [2600:1702:c20:4000:99e4:eab4:3ac3:20b]:41569
More info on the TCP6 sockets in FIN_WAIT2:
ss -o state fin-wait-2 -i
Netid Recv-Q Send-Q Local Address:Port Peer Address:Port
icmp6 0 0 *%eth0:ipv6-icmp *:*
tcp 0 0 [2600:3c03::f03c:91ff:fe2b:6d2c]:https [2605:8d80:6a3:12fa:b41a:1e1f:8206:2e6b]:38628
ts sack cubic wscale:8,7 rto:380 rtt:91.53/71.891 ato:40 mss:1358 pmtu:1500 rcvmss:536 advmss:1428 cwnd:10 ssthresh:36 bytes_acked:307501 bytes_received:507 segs_out:234 segs_in:148 data_segs_out:231 data_segs_in:3 send 1.2Mbps lastsnd:5399472 lastrcv:5430016 lastack:5399148 pacing_rate 2.4Mbps delivery_rate 7.3Mbps busy:1344ms retrans:0/2 rcv_space:28560 rcv_ssthresh:30704 minrtt:40.768
tcp 0 0 [2600:3c03::f03c:91ff:fe2b:6d2c]:https [2600:1702:c20:4000:99e4:eab4:3ac3:20b]:41564
ts sack cubic wscale:8,7 rto:340 rtt:122.521/53.495 ato:40 mss:1428 pmtu:1500 rcvmss:536 advmss:1428 cwnd:10 bytes_acked:18347 bytes_received:450 segs_out:20 segs_in:17 data_segs_out:16 data_segs_in:3 send 932.4Kbps lastsnd:1748236 lastrcv:1778412 lastack:1747932 pacing_rate 1.9Mbps delivery_rate 1.1Mbps app_limited busy:596ms retrans:0/1 rcv_space:28560 rcv_ssthresh:30704 minrtt:93.334
tcp 0 0 [2600:3c03::f03c:91ff:fe2b:6d2c]:https [2605:e000:8563:8500:ac30:3ea6:3d46:b64d]:49358
ts sack cubic wscale:6,7 rto:10664 rtt:2847.83/1159.78 ato:40 mss:1428 pmtu:1500 rcvmss:536 advmss:1428 cwnd:11 ssthresh:11 bytes_acked:307501 bytes_received:471 segs_out:243 segs_in:147 data_segs_out:240 data_segs_in:5 send 44.1Kbps lastsnd:24252 lastrcv:54268 lastack:21092 pacing_rate 52.9Kbps delivery_rate 40.5Kbps busy:41324ms retrans:0/23 rcv_space:28560 rcv_ssthresh:30704 minrtt:78
tcp 0 0 [2600:3c03::f03c:91ff:fe2b:6d2c]:https [2601:184:4780:5002:ad15:cfc4:806f:2080]:37073
ts sack cubic wscale:8,7 rto:232 rtt:31.933/23.886 ato:40 mss:1428 pmtu:1500 rcvmss:536 advmss:1428 cwnd:10 bytes_acked:307501 bytes_received:511 segs_out:221 segs_in:23 data_segs_out:218 data_segs_in:3 send 3.6Mbps lastsnd:1635896 lastrcv:1665944 lastack:1635784 pacing_rate 7.2Mbps delivery_rate 38.5Mbps busy:260ms retrans:0/1 rcv_space:28560 rcv_ssthresh:30704 minrtt:15.989
tcp 0 0 [2600:3c03::f03c:91ff:fe2b:6d2c]:https [2001:1c00:1f09:d500:ec05:daea:ad3:1821]:58338
sack cubic wscale:3,7 rto:320 rtt:119.624/25.914 ato:40 mss:1440 pmtu:1500 rcvmss:536 advmss:1440 cwnd:10 bytes_acked:29815 bytes_received:969 segs_out:26 segs_in:21 data_segs_out:24 data_segs_in:4 send 963.0Kbps lastsnd:81380 lastrcv:111384 lastack:81284 pacing_rate 1.9Mbps delivery_rate 901.4Kbps busy:668ms rcv_space:28800 rcv_ssthresh:32016 minrtt:91.506
tcp 0 0 [2600:3c03::f03c:91ff:fe2b:6d2c]:https [2600:6c64:7b7f:fa70:bc93:ad8f:c665:293]:47578
ts sack cubic wscale:8,7 rto:500 rtt:110.551/97.205 ato:40 mss:1428 pmtu:1500 rcvmss:536 advmss:1428 cwnd:10 ssthresh:102 bytes_acked:307013 bytes_received:461 segs_out:227 segs_in:100 data_segs_out:224 data_segs_in:3 send 1.0Mbps lastsnd:4608160 lastrcv:4638268 lastack:4607796 pacing_rate 2.1Mbps delivery_rate 14.4Mbps busy:692ms retrans:0/6 rcv_space:28560 rcv_ssthresh:30704 minrtt:33.775
tcp 0 0 [2600:3c03::f03c:91ff:fe2b:6d2c]:https [2601:18a:c100:7360:54d0:d99e:2cbf:684]:38931
ts sack cubic wscale:8,7 rto:252 rtt:51.206/31.437 ato:40 mss:1428 pmtu:1500 rcvmss:536 advmss:1428 cwnd:10 bytes_acked:85854 bytes_received:501 segs_out:66 segs_in:15 data_segs_out:63 data_segs_in:3 send 2.2Mbps lastsnd:4589284 lastrcv:4619288 lastack:4589152 pacing_rate 4.5Mbps delivery_rate 4.4Mbps busy:388ms retrans:0/1 rcv_space:28560 rcv_ssthresh:30704 minrtt:24
tcp 0 0 [2600:3c03::f03c:91ff:fe2b:6d2c]:https [2001:1c00:1f09:d500:ec05:daea:ad3:1821]:58206
sack cubic wscale:3,7 rto:300 rtt:99.155/19.576 ato:40 mss:1440 pmtu:1500 rcvmss:536 advmss:1440 cwnd:10 bytes_acked:29815 bytes_received:969 segs_out:26 segs_in:20 data_segs_out:24 data_segs_in:4 send 1.2Mbps lastsnd:1894024 lastrcv:1924032 lastack:1893892 pacing_rate 2.3Mbps delivery_rate 1.5Mbps busy:568ms rcv_space:28800 rcv_ssthresh:32016 minrtt:85.738
tcp 0 0 [2600:3c03::f03c:91ff:fe2b:6d2c]:https [2601:282:201:f0fb:ad90:56a3:2c47:7ecb]:45287
ts sack cubic wscale:8,7 rto:312 rtt:87.997/55.232 ato:40 mss:1428 pmtu:1500 rcvmss:536 advmss:1428 cwnd:10 bytes_acked:42649 bytes_received:519 segs_out:36 segs_in:16 data_segs_out:33 data_segs_in:3 send 1.3Mbps lastsnd:152588 lastrcv:182596 lastack:152392 pacing_rate 2.6Mbps delivery_rate 2.1Mbps busy:456ms retrans:0/1 rcv_space:28560 rcv_ssthresh:30704 minrtt:53.846
tcp 0 0 [2600:3c03::f03c:91ff:fe2b:6d2c]:https [2001:470:1f1b:321:5da3:b731:f1d4:b3f2]:37391
ts sack cubic wscale:8,7 rto:328 rtt:125.015/31.842 ato:40 mss:1208 pmtu:1500 rcvmss:536 advmss:1428 cwnd:2 ssthresh:7 bytes_acked:538637 bytes_received:460 segs_out:459 segs_in:183 data_segs_out:455 data_segs_in:3 send 154.6Kbps lastsnd:620912 lastrcv:677104 lastack:620700 pacing_rate 309.2Kbps delivery_rate 2.8Mbps busy:22432ms retrans:0/7 rcv_space:28560 rcv_ssthresh:30704 minrtt:100.006
tcp 0 0 [2600:3c03::f03c:91ff:fe2b:6d2c]:https [2a01:598:9985:ea8b:bc52:8d92:6316:c50a]:42811
ts sack cubic wscale:9,7 rto:336 rtt:131.35/9.626 ato:40 mss:1338 pmtu:1500 rcvmss:633 advmss:1428 cwnd:10 bytes_acked:58435 bytes_received:2487 segs_out:51 segs_in:51 data_segs_out:47 data_segs_in:5 send 814.9Kbps lastsnd:5063464 lastrcv:5093468 lastack:5063304 pacing_rate 1.6Mbps delivery_rate 1.8Mbps busy:1096ms rcv_rtt:150 rcv_space:28560 rcv_ssthresh:33430 minrtt:114.762
tcp 0 0 [2600:3c03::f03c:91ff:fe2b:6d2c]:https [2001:44b8:6106:5100:5889:9c30:2027:110e]:29833
sack cubic wscale:8,7 rto:556 rtt:310.006/13.442 ato:40 mss:1432 pmtu:1500 rcvmss:536 advmss:1440 cwnd:10 bytes_acked:307501 bytes_received:698 segs_out:225 segs_in:116 data_segs_out:219 data_segs_in:7 send 369.5Kbps lastsnd:197308 lastrcv:227316 lastack:15808 pacing_rate 739.1Kbps delivery_rate 2.6Mbps busy:2728ms rwnd_limited:532ms(19.5%) rcv_space:28800 rcv_ssthresh:30944 minrtt:294.221
tcp 0 0 [2600:3c03::f03c:91ff:fe2b:6d2c]:https [2003:6:36f:5444:111a:27aa:156c:eb88]:60941
sack cubic wscale:8,7 rto:336 rtt:128.89/4.4 ato:40 mss:1380 pmtu:1500 rcvmss:536 advmss:1440 cwnd:10 bytes_acked:161942 bytes_received:765 segs_out:123 segs_in:79 data_segs_out:121 data_segs_in:3 send 856.5Kbps lastsnd:15908 lastrcv:45908 lastack:15800 pacing_rate 1.7Mbps delivery_rate 5.0Mbps app_limited busy:776ms rcv_space:28800 rcv_ssthresh:30944 minrtt:108.861
tcp 0 0 [2600:3c03::f03c:91ff:fe2b:6d2c]:https [2001:44b8:6106:5100:5889:9c30:2027:110e]:29834
sack cubic wscale:8,7 rto:936 rtt:310.86/118.547 ato:40 mss:1432 pmtu:1500 rcvmss:536 advmss:1440 cwnd:10 bytes_acked:1 segs_out:5 segs_in:7 data_segs_in:4 send 368.5Kbps lastsnd:227732 lastrcv:227732 lastack:31812 pacing_rate 737.0Kbps delivery_rate 37.6Kbps app_limited busy:304ms rcv_space:28800 rcv_ssthresh:28800 minrtt:305.042
tcp 0 0 [2600:3c03::f03c:91ff:fe2b:6d2c]:https [2601:282:201:f0fb:ad90:56a3:2c47:7ecb]:45315
ts sack cubic wscale:8,7 rto:296 rtt:85.205/39.4 ato:40 mss:1428 pmtu:1500 rcvmss:536 advmss:1428 cwnd:10 bytes_acked:307501 bytes_received:514 segs_out:223 segs_in:22 data_segs_out:219 data_segs_in:3 send 1.3Mbps lastsnd:150416 lastrcv:180564 lastack:150184 pacing_rate 2.7Mbps delivery_rate 11.0Mbps busy:680ms retrans:0/1 rcv_space:28560 rcv_ssthresh:30704 minrtt:49.824
tcp 0 0 [2600:3c03::f03c:91ff:fe2b:6d2c]:https [2606:a000:4f02:1ff0:3836:cbfc:1c31:8fc2]:35790
ts sack cubic wscale:7,7 rto:3472 rtt:415.179/764.112 ato:40 mss:1428 pmtu:1500 rcvmss:536 advmss:1428 cwnd:10 ssthresh:40 bytes_acked:538272 bytes_received:453 segs_out:387 segs_in:147 data_segs_out:384 data_segs_in:3 send 275.2Kbps lastsnd:4426892 lastrcv:4458812 lastack:4424708 pacing_rate 550.3Kbps delivery_rate 12.9Mbps busy:3488ms retrans:0/4 rcv_space:28560 rcv_ssthresh:30704 minrtt:20.494
tcp 0 0 [2600:3c03::f03c:91ff:fe2b:6d2c]:https [2601:282:201:f0fb:ad90:56a3:2c47:7ecb]:45291
ts sack cubic wscale:8,7 rto:284 rtt:81.097/31.705 ato:40 mss:1428 pmtu:1500 rcvmss:536 advmss:1428 cwnd:10 bytes_acked:49562 bytes_received:514 segs_out:39 segs_in:15 data_segs_out:37 data_segs_in:3 send 1.4Mbps lastsnd:152536 lastrcv:182544 lastack:152396 pacing_rate 2.8Mbps delivery_rate 1.5Mbps busy:496ms rcv_space:28560 rcv_ssthresh:30704 minrtt:59.571
tcp 0 0 [2600:3c03::f03c:91ff:fe2b:6d2c]:https [2600:1702:c20:4000:99e4:eab4:3ac3:20b]:41569
ts sack cubic wscale:8,7 rto:320 rtt:117.997/45.499 ato:40 mss:1428 pmtu:1500 rcvmss:536 advmss:1428 cwnd:8 ssthresh:8 bytes_acked:307013 bytes_received:454 segs_out:232 segs_in:171 data_segs_out:230 data_segs_in:3 send 774.5Kbps lastsnd:1747228 lastrcv:1777408 lastack:1747032 pacing_rate 929.4Kbps delivery_rate 939.7Kbps busy:3192ms retrans:0/12 rcv_space:28560 rcv_ssthresh:30704 minrtt:88.105
I am going to disable IPv6 for a few hours and see if this resolves the FIN_WAIT2 tcp6 socket leak.
Yup, it seems that disabling IPv6 prevents the tcp6 socket leak in Docker (see attached screenshot).
For the time being, I've deleted all changelog.com AAAA records until this is addressed in Docker. I'm open to alternatives, but for now not having IPv6 support seems preferable to leaking TCP sockets.
Since a node balancer is now used, could this change be attempted again? Node balancers proxy traffic to servers over IPv4 so the IPv6 socket leak shouldn't occur.
As for cdn.changelog.com it seems it should be enough to change the CNAME from changelog.map.fastly.net to dualstack.changelog.map.fastly.net. Fastly also proxies connections to the backend over IPv4.
Yes, definitely. Feel free to have a go at it. I'll pick it up as soon as I'm done with the few things that I have in flight.
Didn't realize stuff's now in Terraform – neat! Had a go at it with #283.
cdn.changelog.com doesn't seem to be in Terraform though.
Yup, cdn.* is Fastly. The new infrastructure is covered in detail in The new changelog.com setup for 2019 😉
Will make the necessary changes in Fastly before I close this.
Awesome, thanks!
A few more things:
- [ ] Images on the blog are from
changelog-assets.s3.amazonaws.com. Usechangelog-assets.s3.dualstack.us-east-1.amazonaws.cominstead for IPv6 support.
~~IPv6 on thechangelog.com and nightly.changelog.com?~~
Nice catches, will look into them next time I'm around 👍
I was also going to add some end-to-end tests for this. Do you want to help out with a preliminary PR? You can use proxy.bats as a starting point. This is invoked via proxy-test make target.
@gerhard My bash knowledge is non-existant and I don't have a Linux machine on hand, so I'm afraid I'm unable to help with the tests.
nightly.changelog.com is a legacy hostname which redirects to https://changelog.com/nightly. Where do you see this hostname being used?
I was not even aware that thechangelog.com existed. Is this being used anywhere? This is simply redirecting to https://changelog.com.
@gerhard https://changelog.com/nightly has a link pointing to http://nightly.changelog.com/2016/10/06/. Not sure how I stumbled upon thechangelog.com.
@jerodsanto if Nightly is still a thing, maybe we should move the content example to changelog.com. nightly.changelog.com is completely off my radar.
@tambry I've asked Fastly to make the IPv6 change to cdn.changelog.com, still waiting.
Nightly is still a thing. Here is yesterday's issue:
http://nightly.changelog.com/2019/04/10/
We send this url over to Campaign Monitor to deliver the email.
Are we running nightly.changelog.com on DigitalOcean? If yes, is it worth adding IPv6 to it?
We might also want to update the SSL certificate:
openssl s_client -showcerts -servername nightly.changelog.com -connect nightly.changelog.com:443
...
verify return:1
depth=0 OU = Domain Control Validated, OU = EssentialSSL Wildcard, CN = *.changelog.com
verify error:num=10:certificate has expired
notAfter=May 19 23:59:59 2017 GMT
...
nightly.changelog.com is on DO. Their guide for enabling IPv6 (unfortunately not enabled by default unlike Linode).
One step closer:
dig cdn.changelog.com A +short
dualstack.changelog.map.fastly.net.
151.101.17.162
^A^AAAA
dig cdn.changelog.com AAAA +short
dualstack.changelog.map.fastly.net.
2a04:4e42:4::418
In the interest of wrapping this issue up - it has been open long enough! - I have reduced the scope and shipped it:
I don't expect a response @tambry, but just in case this is still important to you, feel free to open the next IPv6 issue that you are still facing. Rather than creating a single issue which is likely to take a really long time to address, smaller ones that we can tackle between Kaizens are more likely to get us to full IPv6 support sooner. Thanks for sticking with us!
@gerhard Thank you very much for finally getting around to this!
I clicked around a bit and reported #419 and #420. They're smaller and hopefully fairly contained, so should make it easier to get the dopamine rushes for solving them. 😉