sites-using-cloudflare
sites-using-cloudflare copied to clipboard
pirate
Some sites may be using the CloudFlare SSL proxy without using CloudFlare DNS
"Also there are some sites that are using the CloudFlare SSL proxy without using CloudFlare DNS -- for example betterment.com is using Amazon Route 53 and is not in sorted_unique_cf.txt . Basically I think you just have to make HTTPS requests and look for the CF-Ray or Server: cloudflare-nginx header. It probably would be useful for someone to write a script to make requests to the domains in sorted_unique_cf.txt and update it to remove the ones that aren't using the SSL proxy." @youngj
This might not be as easy as it sounds, CF will probably start to rate-limit the source IP if someone decides to curl every page on that list, providing inaccurate/incomplete results
As long as you're requesting HEAD only (aka curl -I) with a throttled pool it shouldn't be an issue.
I could probably write up a little node.js script to automate the process.
I don't think the request / response size matters. What i think is that CF starts to throttle or bail out completely if you bombard them with requests. In those scenarios, most often quantity matters - not size
I wrote a Go script to check a list of domains for the CF-Ray header: https://github.com/youngj/sites-using-cloudflare/blob/master/check_domains.go . I don't have time to actually filter the list of domains now, but perhaps someone else can use it.
Domains using cloudflare's reverse proxy should resolve to a cloudflare IP because the proxy's front end is owned by cloudflare. You'll probably get really accurate results by just resolving the domain names and checking them against ASN blocks to see if they're cloudflare ip's.
@jrruwe that works, but unfortunately lots of people might be turning off cloudflare today, so the data will soon be out of date.
I wonder if reverse dnsing the ip would return the old dns name even after the forward dns records have been changed??
I wrote my own Node.js shellscript and will begin testing all the domains that start with 8 as a preliminary run, since there is only 11599 of them.