imgbrd-grabber icon indicating copy to clipboard operation
imgbrd-grabber copied to clipboard

Published 20 hours ago • verified publisher icon Bionus

Integration with FlareSolverr/other Cloudflare solvers

Open aaabbbccc16 opened this issue 2 years ago • 3 comments

Is your feature request related to a problem? Please describe

At least 2 boorus that I know (furry.booru.org and rule34hentai.net) have the new Cloudflare challenge that can only be solved with an actual browser. It's not much, but more may appear in the future, so it would be reasonable to start considering solutions for this problem. In addition, whenever grabber encounters this protection, it displays a generic "Possible reasons: server offline" message that is not very helpful to understanding what the actual source of the problem is and how the user can deal with it. Currently, to bypass the new challenge it is necessary to open the site in a browser, wait for it to solve the challenge and copy over the User-Agent and cf_clearance cookie from the developer console into the source's settings, which will allow the user to browse the site for a day or so.

Describe the solution you'd like

Jackett deals with the same problem by using FlareSoverr proxy that uses a headless browser to solve the captcha. Although, from the README's wording it sounds like FlareSolverr proxies all the incoming requests through the browser instead of just passing the clearance cookie back, which would be a bit of a problem for the amount of requests Grabber normally makes. I would very much like to see this or similar automated solution being integrated into Grabber.

Describe alternatives you've considered

It that is not feasible, I would like to see some semblance of semi-automation for manually retrieving the clearance cookie from the browser, or at least a separate error message for Cloudflare challenge related errors with information on how to manually retrieve the cookie from the site.

aaabbbccc16 avatar Apr 08 '22 00:04 aaabbbccc16

Other cloudflare scrapers mentioned here: https://github.com/Bionus/imgbrd-grabber/issues/1858#issuecomment-573377447

aaabbbccc16 avatar Apr 09 '22 14:04 aaabbbccc16

Rather than implementing a solver, isn't it more reliable to have a built-in browser open so that the user can "solve" the Cloudflare challenge?

Bionus avatar Apr 09 '22 16:04 Bionus

Well, really, I meant "any way of (semi-)automatically retrieving the clearance cookie" when I said "solver".

aaabbbccc16 avatar Apr 09 '22 17:04 aaabbbccc16