SnapshillBot
SnapshillBot copied to clipboard
justcool393
Refactoring to prepare for gevent
This is a WIP branch where I'm refactoring a bit. We currently perform all http requests sequential, which takes a lot of time per submission and limits our ability to add more subreddits to monitor. By using gevent (or some other solution) we can perform the requests (mostly) in parallel, or at least make them non-blocking. This hopefully speeds up the bot a huge deal.
@justcool393 ok, so I think I got everything worked out. I'm performing all archiving actions in parallel (it's wicked fast 🐎💨), but apply rate limiting to the creation of archives of links pointing to reddit (I know this sounds weird).
I also noticed we got banned from 4 subs, some of which are quite active, so I decided to automatically unsubscribe from subs we have been banned from.
If you're ok with this, I'm going to test this new setup on the server.
@hidde-jan I don't know how I missed this message. This is excellent.
Small thing, I believe Archive.is automagically ratelimits its own requests to reddit (ceddit is based of some quirk on how reddit works, so those are based off of the users who are browsing the archive), so it may only be necessary for sites that do not ratelimit themselves. I do think this is an great idea. The reason I had it at five initially was because reddit was weird about requests when they came from archive.org.
Further, I send a message to the admins about maybe getting one of the services (that isn't currently in place) un-spamfiltered for more redundancy. Also, I'll ask again about archive.org when I get a response.
I'm still not completely satisfied. It currently only handles one submission at a time. I want to set up some queue based system that can handle multiple things at once. The xkcd transcriber bot has something like this. I've already started looking at it.
Ok, I think I know why I abandoned this last year. Praw is not thread safe. It's rate limit function depends on time.sleep and even which patching that out means that there is no actual way to get this working. I'm still pretty happy with the refactoring in this PR, so I'm going to port them to the master brach and close this PR.