gotosocial
gotosocial copied to clipboard
superseriousbusiness
[bug] Pleroma instances tend to massively overload my GtS instance for some reason
Describe the bug with a clear and concise description of what the bug is.
Due to some sort of incompatibility (perhaps #107), Pleroma instances tend to get stuck in a loop requesting the same things over and over from my instance. When making a particularly popular post, the effect of lots of Pleromas doing this causes CPU load to spike to ~200%, the journal to get overwhelmed, ... -- basically a small Denial of Service attack.
What's your GoToSocial Version?
v0.10.0 git-2470e21 (patched)
GoToSocial Arch
amd64 binary
What happened?
- Make a post that gets boosted by a bunch of people
- Notice GtS getting slower to respond
- Have my Prometheus alerts tell me that the server is overheating and has its resources stretched
- Check the logs to find lots of Pleroma spam
A random sample of 100 lines of it looks like this.
What you expected to happen?
Although this probably needs to be fixed pleroma-side, it'd be nice to have some mitigation for this on the GtS end -- I currently just block the pleroma User-Agent entirely, but some rate-limiting to detect this case might be useful
How to reproduce it?
No response
Anything else we need to know?
No response
@tsmethurst I just had a thought, what if we are also doing the same thing we found that some Mastodon instances were doing and serving self-referential status collection pages?
this would be very ironic :')
I'm pretty sure this one is actually PEBKAC -- I had hacked up my GtS database to move some of my old posts over from another instance, but this resulted in the pinned toot being broken, and I think that's why the Pleromas carried on crashing and retrying in a big loop. Whoops! Apologies for the false alarm >.<
I lied! I pinned a non-cursed toot, left it a while, and came back to another sea of constantly retrying Pleromas in the logs :c
Hmmm.... could it be because you've got your domain (eta.st) serving webfinger responses directly rather than forwarding them to your GtS server? It does look like you're serving the correct data, but it might be confusing Pleroma specifically for whatever reason.
It seems very linked to the pinned toot -- unpinning the toot will make the Pleromas stop looping, for example -- so I doubt that's it
Does it persist if you have multiple toots pinned? Is it specifically with one pinned toot that it occurs?
I just updated my own patchset to HEAD actually, so I'll pin a toot and see if I can get some pleromas stuck in it again...
Update: It's 2 days later and I just opened my gts logs to find about 10 Pleroma requests per second, so this is still an issue! Please let me know if there's anything more I can do to debug this :)
Alright, thanks for checking it out! I'll see if I can figure out just what it is about GtS pinned posts that pleroma doesn't like...
If it also affects Akkoma, I could try to follow and take a look into the logs.
It only affects sufficiently old Akkoma; modern Akkoma versions don't seem to have an issue.
Ah really? So it's something that Akkoma identified as a bug recently and fixed?
Just a thought, btw, are you pinning toots that you 'imported' and then fiddled with in the database?
That was originally the case, but the problem exists even with non-'imported' toots.
I'm still not sure what to do with this one. Is it still happening or have the 'omas fixed it on their side?
Very much so -- pinned a post today and came back an hour later to find GtS using all the CPU on the machine :(
How about now? Just want to check if any of the code updates over the last 5 months have resolved this. Otherwise I'll close it and just chalk it up as an *oma bug, because I really don't think it's anything we're doing :thinking:
Seems fixed! I've had a pinned post for 2 days and haven't seen the problem recur; additionally, the handy new HTTP header block thing makes this easy to mitigate without having to recompile, so closing this seems like a good idea :)
(whoops, didn't actually mean to instantly close, but I guess that's fine?)