grab-site
grab-site copied to clipboard
Add support for Cloudflare DDoS protection screen
Let's run their JavaScript somehow, or import someone else's solution to the Cloudflare problem.
@ivan Could you explain your proposed solution a bit further? I would like to pick this up.
My first step would be to review the things on https://github.com/search?q=cloudflare+scrape+fork%3Atrue and see if anyone has solved the problem in a satisfactory manner where it can just be imported from grab-site and called in one of the wpull hooks. https://github.com/Anorov/cloudflare-scrape/blob/master/cfscrape/init.py looks promising, but I haven't tested it.
If none of the existing solutions are good, I would look at using https://github.com/microsoft/playwright to visit the cloudflare-protected page, then copy the cloudflare cookie out of the browser and add it to the grab-site cookie jar.