mikf
add support for FlareSolverr?
Perhaps we could get support for FlareSolverr? https://github.com/FlareSolverr/FlareSolverr Could help with bypassing the cloudflare issues and DDoS-Guard issues some people are having by feeding the urls though the api. Maybe im wrong here, but it seems to work wonders for my Jackett instance that scrapes CF protected sites.
FlareSolverr starts a proxy server,
Couldn't you just use this with --proxy, then?
FlareSolverr starts a proxy server,
Couldn't you just use this with
--proxy, then?
I tried that in the past with no success. +1 for this.
yeah, tried setting it up as a proxy for quite a while with no luck at all. would be great if gallery-dl interfaced with flaresolverr directly much like the arr suite and jackett does, just set the path to the hosted flaresolverr client and use the api directly from there. enabling ease of use for everyone.
I'm tinkering with this. The problem is that flaresolverr isn't a proxy, it's just a... well, a flare solver.
You give it a URL, it opens a headless Chrome to solve the cloudflare challenge, and passes back a JSON string with your status, some messages, your cookies and the page returned. Because it runs a headless chrome, it's pretty slow for a proxy. It'd be kind of a poor choice for that.
One thing I'm playing with is making a local proxy wrapper. If there's a cloudflare challenge, it calls flaresolverr and updates your gallery-dl.conf. If it doesn't detect a challenge, it just proxies your request.
Hopefully more to come, but post if anyone has other thoughts or experiments.
OK, maybe before I go too far down this rabbit hole, let me ask @mikf - Where/how would you want to see this implemented?
I can keep going on this proxy script, but it seems like that's kind of a lot of work for some of your userbase.
We can also build it into individual extractors. Here's a working proof of concept.
Or... We could add it to Common (or a separate file, whatever) to be used by multiple extractors. I could give that a shot, but don't want to be so bold as to assume that's the direction you'd want to take it.
Here's proof of concept for what it would look like incorporated into common.py. Of course we'd want more testing before going too far with it, but it has the benefit of being a one-line modification to extractors to support Flaresolverr.
Oh, I also had to add the Flaresolverr URL to my gallery-dl.conf:
"feetplaza": {
"flaresolverr": "http://127.0.0.1:8191"
},