grab-site icon indicating copy to clipboard operation
grab-site copied to clipboard

Add support for Cloudflare DDoS protection screen

Open ivan opened this issue 7 years ago • 2 comments

Let's run their JavaScript somehow, or import someone else's solution to the Cloudflare problem.

ivan avatar Dec 07 '17 19:12 ivan

@ivan Could you explain your proposed solution a bit further? I would like to pick this up.

gabefair avatar Apr 19 '20 20:04 gabefair

My first step would be to review the things on https://github.com/search?q=cloudflare+scrape+fork%3Atrue and see if anyone has solved the problem in a satisfactory manner where it can just be imported from grab-site and called in one of the wpull hooks. https://github.com/Anorov/cloudflare-scrape/blob/master/cfscrape/init.py looks promising, but I haven't tested it.

If none of the existing solutions are good, I would look at using https://github.com/microsoft/playwright to visit the cloudflare-protected page, then copy the cloudflare cookie out of the browser and add it to the grab-site cookie jar.

ivan avatar Apr 19 '20 21:04 ivan