ShokoAnime
Strict and Loose AniDB Rate Limiting
Okay, so there are people with sad, sad internet. When your ping to AniDB is over 300ms, then there's no telling how the packets are going to arrive.
My proposition is this: A config to increase the rate limiter timing Loose - as described by AniDB (current) Strict - Add 300ms to each request for more of a buffer* Stricter - Add 500ms to each request*
This should help in most cases where bad ping is the cause of bans. It also allows for an extra troubleshooting step so people don't need to wait for a dev to look into it, as bans are hard to debug.
*When pulling for extended time, and the limit is increased, we double this value as well.
The way AniDB UDP API is implemented it is essentially same as HTTP as there is no sequence id or anything so you can't send it requests out of order, you have to wait for response, so I am not sure how the situation you are describing would occur.
Because people with shitty internet can't predict what happens between them and the target
Why does it matter if you still wait for a response ? We only have one active request at a time and we guarantee specified time between them. Your theory implies we have multiple active requests, which we don't, with only one active request it is not possible for them to arrive sooner that specified timeout only later.
Well for UDP you can send requests 2s apart, then they get throttled at ISP switches, and eventually 3 requests show up on the other side of the world at the same time
Yes that is correct but there are multiple reasons why this does not apply to our code and AniDB. AniDB API has no support for out-of-order requests, when you use TCP that is done by TCP stack for you but with UDP you need to implement it into protocotl itself.
AniDB API has no request id, so if we would send multiple requests we would get errors when responses get mixed up, for example addmylist getting response from mylist status. Therefore we never send multiple requests at once, that is why your situation with ISP switches and delays is impossible, there is never more that one request out, and we wait for response.
We wait for quite a few seconds so it is highly unlikely for a packet to be delayed that long, UDP packets don't live minutes in wires.
I am guessing you are referring to HTTP bans, but in those cases it is out of our control since AniDB does not want to disclose their logic to make it harder to circumvent and also due to the fact that they do not authenticate HTTP requests so they limit by IP or even IP range so if you got abusive neighbor you get banned. I remember someone mentioned in discord that images are affecting rate limit in a bad way so if we put everything (UDP, HTTP and images) into same limiter it will become super slow but I am pretty sure there will be almost no bans.
I did put images into the same limiter. It waits, downloads image batches, waits, then does other stuff. On the plus side, no one in dailies has complained for quite a while.
As far as I remember it only limits batch not every image and since you added seyuu's and characters there were more reports on bans, so if we limit each image separately there should be a lot less bans. It will become super slow tho.
I've not seen any issues with images lately. We could have a separate limiter that only rate limits 0.2s or something for those batches. That's more than reasonable for a browser.
Any info on that? Because it makes no sense to limit everything else except images. We might want to check with AniDB staff.
It makes perfect sense. Images don't go through the API. They are a part of the website. You request images in the same way that a browser would.
Except it's a fluke that all the time when http bans occur it's because a few images are occuring between anidb http requests?
On Mon, Apr 23, 2018, 1:11 PM da3dsoul [email protected] wrote:
It makes perfect sense. Images don't go through the API. They are a part of the website.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ShokoAnime/ShokoServer/issues/699#issuecomment-383442432, or mute the thread https://github.com/notifications/unsubscribe-auth/AA8rpkVa4KpzU48R70bX6a8oHqy-pz4tks5trUZ9gaJpZM4Rl_Z0 .
Problem is here https://github.com/ShokoAnime/ShokoServer/blob/b36b92e2585e3a2084c46169e3aad1361dfeeef0/Shoko.Server/Commands/Import/CommandRequest_DownloadImage.cs#L258 We limit the command itself but then it goes and rapidly downloads a ton of images potentially.
I remember someone saying that images do not trigger the ban but they count towards the request count so if you download a bunch of images and then do an HTTP API request you get banned.
here is a quote from CDB-Man I found in support
if using a web browser, iirc the limiter is something like 7 page loads in 10s max, with the decay being slower than that
so images fall under browser category I guess, it would explain bans when you download seyuus and characters as there is usually 5+ but not the series images.
Ah first I've heard of the 7 in 10s thing. The 0.2s thing would work in that case
