SuperGouge
No longer downloading images
CTW has been working great with 4Chan for years, but some time in late April or early May of 2024 it is no longer downloading 4Chan images. Everything seems to be working fine, except under the STATUS column, it says "Downloading Images: 0 of ??? completed" where ??? is replaced with the number of files in that relevant link. When I look in the appropriate download folder, I see the HTML file, the THUMBS folder which is full of thumbnail files, but the 'actual' images aren't downloaded.
did it resolve for you? was the image server down? it worked for me until yesterday.
did it resolve for you? was the image server down? it worked for me until yesterday.
Nope. Still not working at all.
For me it even stopped downloading the thumbnail folder. Folder of the thread gets created, but nothing downloaded. I tried every old version of CTW i could find, used the administrator mode and it just wont download. I also cant find a proper alternative that keeps crawling the threads like CTW did. I hope someone way smarter than me can figure the issue out.
I stumbled into this issue as well this weekend. No idea why it is not downloading any images or threads but I imagine that some change in 4chan's backend has broken ctw.
I changed the user agent in CTW which solved the issue for some weeks but about 3 days ago it stopped downloading anything. No images, no thumbnails, no thread HTML.
I would like to solve the issue if I could since it's the best softeware I've ever used. My all time favorite softeware I always use saving time.
Please could someone with knowledge fix this issue? Let's start a donation action for the saver.
UPDATE
I tried ChatGPT to fix the problem but still can't fix it. Please post any solution.
Anons at 4chan's /g/ had a stab at this and seems to have figured out that the problem lies in that Cloudflare's bot protection walls off scraping of the board. Since CTW can't get past it automatically. Suggested solutions were, among the few, swapping to using API or using Clodflare's cookie grabbed from a browser to validate the session. Someone even posted a snippet of code which supposedly solves the issue but requires manually adding your own Cloudflare cookie. I've tried it myself but didn't have any luck. Perhaps someone more talented than me can figure it out.
The thread has been since archived and soon will hit 404, so here's external archive: https://desuarchive.org/g/thread/104789248
For simplicity, here's the copy-paste of the provided code snippet. I claim no rights to this as I'm not the one who came up with it. Godspeed.
var cookieContainer = new CookieContainer(); var cookie = new Cookie("cf_clearance",#VALUE#, "/", ".4chan.org"); cookie.Expires = new DateTime("2222222222"); cookieContainer.add(cookie); request.CookieContainer = cookieContainer;>Add this to the BuildHttpWebRequest method in General.cs and replace #VALUE# with the value of the cf_clearance cookie in your browser I set the cookie expiry for about 15 years, no idea if Cloudflare will care but I figure it can't hurt. Proper session management means controlling session and cookie duration server-side but I dunno if that's how they've implemented it since I can imagine it would make things a fair bit more resource intensive for little to no benefit
Thank you for the info. I am a noob but I am assuming that
Add this to the BuildHttpWebRequest method in General.cs
means that I have to compile the source code afterwards? Like I said, I'm a noob.
Also, changing the program to use the API would require a major rewrite, since - from what I read in the thread - CTW currently scrapes off of the HTML.. Can ChatGPT rewrite the relevant portions of the code?
Indeed, since CTW is written in C#, the code needs to be compiled to make it work. Notably, the actual method that the anon from /g/ meant is BuildWebRequest rather than BuildHttpWebRequest, I presume.
As for refactoring the application to use API, I have no idea if ChatGPT would manage to do this successfully. Always worth a try in absence of better solution I guess. I was focusing on trying it to get me to help with the cookie solution but it doesn't work as it should in the end still.
One anon in the thread claims he got it to work, though we never know it this is actually true.
I was focusing on trying it to get me to help with the cookie solution but it doesn't work as it should in the end still.
Does your request get through at all? I mean does it pass the challenge? Btw, I left CTW running and last week it got through twice, at random times. It downloaded the HTML and images like normal. Not sure if this is helpful information.
Unfortunately it seems my requests are not getting through spent some more time tinkering with it today but with no luck.
I left CTW running and last week it got through twice, at random times. It downloaded the HTML and images like normal. Not sure if this is helpful information.
That is helpful, thank you. I imagine the Cloudflare's bot protection is either imperfect or at times it just doesn't act up to block CTW. Or there are some other shenanigans at work.
I used to fix it for a week or so using my browser agent string, but it stopped working again.
does anyone have any other ideas? or times when the bot protection was more permissive?
This is only for 4chan, but I asked free Grok (and other AI overlords) to make me a thing (it's a chrome extension). It's vibe coded, as the kids say.
I am sharing, since it reached a state where it might be useful for someone, maybe. I'm not a programmer, use at your own risk.
https://github.com/BurrowBird/4chan-Thread-Muncher
[edit]: Updated the extension a bit recently to make it more useful (also AI is getting better at coding).
@BurrowBird I have been using ychan and ctw for years. Ychan went out first now ctw. Was lurking this thread for awhile. That thread muncher 9000 actually works and works better than most of the other extension alternatives. If this is vibe coding keep it up this could be the next big chan tool,
just restarted CTW to download from one of the archives and lo and behold,
IT'S WORKING AGAIN!
just restarted CTW to download from one of the archives and lo and behold,
IT'S WORKING AGAIN!
Not working for me.