FanFicFare
FanFicFare copied to clipboard
JimmXinu
Cloudflare and FanFiction.net doing its thing again
FanFicFare 4.3.0, Calibre running from source.
Attempting to update from FFNet gives out a Cloudflare 2 error. Several other users have reported this on MobileRead. Cache works as expected. Checked Twitter and there's no indication that FictionPress is working on an update, but I will keep monitoring.
Not much to be done about it (there doesn't seem to be any recent updates to the scraper), but at least here's a report to monitor the issue.
I don't know if it is the same thing but I get the following error when using CLI:
cloudscraper.exceptions.CloudflareCaptchaProvider: Cloudflare Captcha detected, unfortunately you haven't loaded an anti Captcha provider correctly via the 'captcha' parameter.
During handling of the above exception, another exception occurred:
fanficfare.exceptions.FailedToDownload: cloudscraper reports: "Cloudflare Captcha detected, unfortunately you haven't loaded an anti Captcha provider correctly via the 'captcha' parameter."
It looks like there is a change in cloudscraper's use and it needs a new option to be set. Or maybe it's an old option and now it raises an error... I don't know
It looks like ffnet has increased their Cloudflare blocking level again.
As far as I know, there isn't anything more we can do in FFF than there was the last time this happened.
I suggest reviewing these resources:
FAQ: https://github.com/JimmXinu/FanFicFare/wiki/FAQs#why-am-i-having-errors-downloading-from-fanfictionnet--why-am-i-getting-cloudflare-errors-downloading-from-fanfictionnet
Browser Cache Feature: https://github.com/JimmXinu/FanFicFare/wiki/BrowserCacheFeature
Browser Proxy (third party): https://github.com/nsapa/fanfictionnet_ff_proxy
@JimmXinu We can use puppeteer though? I just tested my https://github.com/NightMachinary/.shells/blob/master/scripts/javascript/curlfull.js , and it worked fine. Using the browser cache is not a very good solution (I use fanficfare on a server).
@JimmXinu A general workaround is to let the user supply a downloader, like youtube-dl's --external-downloader, if you don't like bundling puppeteer.
First off, I'm not doing anything about this in a hurry. More than once, we've seen ffnet's CF level go up for a while and then go back down after a while--think days, not hours.
Second, Cloudflare is a service specifically for detecting and blocking automated access. I assume (and past evidence agrees) that they can already block well known automated browser tools. If I recall correctly, earlier in the year, we experimented with headless proxy browsers and all of them were blocked after a few requests.
That's why our current generation of work arounds are based on using a live browser.
I was already using Cache... but since I reread the wiki I tried and I downloaded the entire story using WebToEpub and it downloaded the whole story. But when I immediately afterwards used FFF is didn't work
I was already using Cache... but since I reread the wiki I tried and I downloaded the entire story using WebToEpub and it downloaded the whole story. But when I immediately afterwards used FFF is didn't work
Try use_browser_cache_only:true to prevent it falling back to the site. I'm assuming it's some user error on my end, but for whatever reason I have trouble getting it to use the cache if I only set use_browser_cache:true.
Is it worth - if ffn keep this up - investigating how their app bypasses these checks?
Try
use_browser_cache_only:trueto prevent it falling back to the site. I'm assuming it's some user error on my end, but for whatever reason I have trouble getting it to use the cache if I only setuse_browser_cache:true.
This is what I've tried (using FanFicFare 4.3.0 and Firefox 88):
- Download https://www.fanfiction.net/s/13747780/1/ with WebToEpub
- Add
browser_cache_path:/home/matej/.cache/mozilla/firefox/uxi2c7cz.default/cache2to[defaults]inpersonal.ini - Run:
tmp@stitny$ fanficfare -o use_browser_cache_only=true https://www.fanfiction.net/s/13747780/1/
cloudscraper.exceptions.CloudflareChallengeError: Detected a Cloudflare version 2 Captcha challenge, This feature is not available in the opensource (free) version.
During handling of the above exception, another exception occurred:
fanficfare.exceptions.FailedToDownload: cloudscraper reports: "Detected a Cloudflare version 2 Captcha challenge, This feature is not available...."
Something’s wrong. Why even with use_browser_cache_only=true I get Internet connection?
Do you also have use_browser_cache:true under [www.fanfiction.net]? use_browser_cache_only:true doesn't do anything without use_browser_cache:true
Do you also have
use_browser_cache:trueunder[www.fanfiction.net]?use_browser_cache_only:truedoesn't do anything withoutuse_browser_cache:true
Which is a bit weird, isn't it? Anyway, this works.
fanficfare -o use_browser_cache=true -o use_browser_cache_only=true -o use_cloudscraper=false \
https://www.fanfiction.net/s/13747780/1/
I'm assuming it's some user error on my end, but for whatever reason I have trouble getting it to use the cache if I only set
use_browser_cache:true.
Just for posterity, check_next_chapter:true seemed to be the cause of this.
I have also recieved the following errors in the last 24 hours.
Status Title Author Comment URL Error Unknown Unknown cloudscraper reports: "Detected a Cloudflare version 2 Captcha challenge, This feature is not available...." https://www.fanfiction.net/s/5598642/1/ Error Unknown Unknown cloudscraper reports: "Detected a Cloudflare version 2 Captcha challenge, This feature is not available...." https://www.fanfiction.net/s/5244813/1/ Error Unknown Unknown cloudscraper reports: "Detected a Cloudflare version 2 Captcha challenge, This feature is not available...." https://www.fanfiction.net/s/5012016/1/Partners
I have also recieved the following errors in the last 24 hours.
Yes, Cloudflare is acting again, and there is nothing we can do about it. Workaround with WebToEpub is the only thing I know to work.
Is it worth - if ffn keep this up - investigating how their app bypasses these checks?
The author of at least one other 3rd party reader app I know of has already looked at it and states that the code at issue is a totally obfuscated mess that he has no intention of trying to dig into. If you want to try, feel free. But, keep in mind, even if you manage to figure something out, as soon as more than a few people try to use it they'll just update the app and invalidate everything you did.
The owner of FFN is focused on getting his ad revenue, and going to war with a company like Cloudflare is a zero sum fight you aren't going to win in the end. The only thing you can try to do is work around them like Jimm is with the FFF browser cache feature. I just hope FFN doesn't try to implement some form of cache busting technique.
Is it worth - if ffn keep this up - investigating how their app bypasses these checks?
The author of at least one other 3rd party reader app I know of has already looked at it and states that the code at issue is a totally obfuscated mess that he has no intention of trying to dig into. If you want to try, feel free. But, keep in mind, even if you manage to figure something out, as soon as more than a few people try to use it they'll just update the app and invalidate everything you did.
The owner of FFN is focused on getting his ad revenue, and going to war with a company like Cloudflare is a zero sum fight you aren't going to win in the end. The only thing you can try to do is work around them like Jimm is with the FFF browser cache feature. I just hope FFN doesn't try to implement some form of cache busting technique.
The official FFN app doesn't have ads, so whatever reason they have for this can't be ad revenue.
Rather than believing that ffnet has deliberately set out to ruin our collective day, I suspect that they are using Cloudflare for protection from aggressive, automated systems attacking their site.
Unfortunately, individual users with automated download software (IE, us) are extremely similar in appearance from their POV.
I'll add what I said over on MR recently:
I have lost basically all interest in fighting with ffnet about this. Browser cache works well enough for me for the few active authors I still read on ffnet.
Unless someone comes up with a radical new idea, I'm not thinking about it anymore right now.
Many thanks to the members here are willing and able to help each other.
I have lost basically all interest in fighting with ffnet about this. Browser cache works well enough for me for the few active authors I still read on ffnet.
I basically agree with what you said, with one exception (and no I have no idea how to do it): at least checking that the work has new update. I have still 776 FFN stories (yes, trying to move to AO3 as much as possible, but this is the best I have managed so far) and I have no clue how to check that I am not missing on new chapters. Update itself is done easily with browser cache.
If you have an ffnet account, you can go to Account Page > Alerts > Story Alerts and compare the dates of followed stories to your local copies. Setting up Calibre with a last-updated column will help too.
I have a discord bot which has been able to access FFN most of the time recently. Is fanficfare using the latest version of cloudscraper?
On Jun 14, 2021, at 5:14 PM, chocolatechipcats @.***> wrote: If you have an ffnet account, you can go to Account Page > Alerts > Story Alerts and compare the dates of followed stories to your local copies. Setting up Calibre with a last-updated column will help too.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.
Cloudscraper's last update was on April 9, and I believe it's already been incorporated into FFF. Also, the error specifically mentions the version 2 CAPTCHA, which the Cloudscraper developer has behind a paywall (this was discussed in the last thread).
Has anyone even gotten the paywall-d version to work? At this point I'd be willing to shell out a few bucks, but not if it doesn't work with Fanficfare.
Has anyone even gotten the paywall-d version to work? At this point I'd be willing to shell out a few bucks, but not if it doesn't work with Fanficfare.
I haven’t found a paid version. Is there a one?
It was discussed in the last thread: https://github.com/JimmXinu/FanFicFare/issues/622#issuecomment-756479078
The Google Chrome extension: WebToEpub still can download from FF.net.
https://github.com/dteviot/WebToEpub/tree/ExperimentalTabMode
The Google Chrome extension: WebToEpub still can download from FF.net.
Works in Firefox as well. But the problem is that some more sophisticated things FFF can do, this extension cannot (namely, --update-epub).
@mcepl Better than nothing, especially if one can't use that Windows program to download stuff from FFN, but has a Firefox, Chrome or Chrome-based browser.
I run FanFicFare on a headless Raspberry Pi and submit URLs from my phone, so using any cache isn't viable unfortunately. For now, I've just stopped it checking for updates and I'm seriously considering whether it's worth decompiling the app and seeing if the same method it uses is viable (or if using the app cache is viable)...
does any one know if there is a fanfiction.net app that let us read like a book instead of scrooling if so please let me know so ican read my books intill this problem is solved beacse the scrolling on fanfiction.net makes me sick
FanFiction.net has an official app. From what I understand all other ones are blocked.