BlizzardApiReader
BlizzardApiReader copied to clipboard
avivbiton
More types of RateLimiter
We could add more types of rate limiters, it's up to you, everything is acceptable as long as it's not already implemented by someone else.
Does Blizzard return x-ratelimit-reset and x-ratelimit-remaining in the api response headers? If so I could make a rate-limiter that handles what Blizzard returns for accurate and efficient rate-limiting.
Content-Type: application/json;charset=UTF-8 Date: Fri, 14 Sep 2018 07:36:37 GMT Retry-After: 600 Server: Apache Vary: Accept-Encoding X-Apikey-Qps-Allotted: 2 X-Apikey-Qps-Current: 1 X-Apikey-Quota-Allotted: 5000 X-Apikey-Quota-Current: 1 X-Frame-Options: SAMEORIGIN X-Mashery-Responder: prod-j-worker-us-east-1b-55.mashery.com X-Quota-Reset: Saturday, September 15, 2018 12:00:00 AM GMT Content-Length: 2992 Connection: keep-alive
This is an example of response header. You can view headers at https://dev.battle.net/io-docs
First time looking into contributing to a open source project, so please bear with me.
If I am reading the above header response correctly, we basically just need to check if we have reached the qouta allotted for that day?
Yes X-Apikey-Quota-Allotted shows the maximum requests
And X-Apikey-Quota-Current shows the current requests sent
If you need help, feel free to ask
Since it doesn't seems anyone is currently working on this, anyone feel free to comment and grab this issue On a side note: some changes may require to happen in the IRateLimiter interface in order to make it happen so feel free to do just that
I've been going over the rate limits on the blizzard site. I'm sure i'm missing something but it looks like all the rate limits are time based, like x calls per day, per hour. Wouldn't the existing time based limiter cover all those cases?
It's a bit different, using the data in the response header will provide the limit for the specific key.
The current limiter doesn't use that, it just store it's own limit and use that instead. it is more useful when you have like many clients using the same key and you want to lower and limit the requests rate for each client.
Hope that makes sense
That makes sense now. Basically if you have key 1234 and 456. If key 1234 has made 5 calls in the last hour and can make 10 calls per hour, then that key can only make 5 more calls. If 456 has made none then it can make 10, assuming the response header says both of there limits are 10 calls per hour. I haven't wrote any code for this, so if someone has started on this don't worry about crossing over each other.
@markfila Are you still working on this? I might hop in if no one else does
@owlstack i havent done anything with it, feel free to work on it. I started on the starcraft 2 api models and i am still working on that, and wont have time work to work on anything else.
@owlstack My original thought when looking at this was to have a dictionary where the key was the api key and the lookup value was a class that checked the number of queries per second/hour, using the max values passed back from blizzard to set the limits. In the api reader constructor we would populate the dictionary using the list of api keys in the api config, and just check like api reader is doing now if any key has not exceeded its rate limit. I'd be curious if we come up to the same logic, if you implement this.
That's out of the scope of rate limiter interface. But if you would implement something like this, you would use a rate limiter just to check the limit not to sure keys and things like that
Hi avivbiton, I may have some time to look at this.
What is the preferred method of adding rate limiters by the end user? Also are modifications to ApiReponse.cs OK to gain access to the headers?
Hey sorry for the late response @oisinmcl
limiters.add is the preferred way and feel free to make any modifications, we'll review them anyway. I'm here if you need anything (even though it has been years since I touched this project)
Ok @avivbiton I am new to open source so will try my best with this. Give me a few days to a week I only have limited time (family)
@avivbiton I have a question in relation to the limiter described above. I am new to the blizzard API so are unsure how it used to work however it looks like it has changed now, in the docs here: https://develop.battle.net/documentation/guides/getting-started it states the following for throttling:
Throttling
API clients are limited to 36,000 requests per hour at a rate of 100 requests per second. Exceeding the hourly quota results in slower service until traffic decreases. Exceeding the per-second limit results in a 429 error for the remainder of the second until the quota refreshes.
We could do a limiter based on this per client I suppose I guess but then I don't see why the end user would not just use the current one and set the limit per hour and limit per second. We could create a rate limiter silently that enforces these per second and per hour limits if you wish
Thinking back based on what you say above you would said it would be nice for limits to be based per client. So we could implement the time based limits per client (passing a client id or something). This makes other clients that are not throttled still free to make requests
@avivbiton I have a question in relation to the limiter described above. I am new to the blizzard API so are unsure how it used to work however it looks like it has changed now, in the docs here: https://develop.battle.net/documentation/guides/getting-started it states the following for throttling:
Throttling
API clients are limited to 36,000 requests per hour at a rate of 100 requests per second. Exceeding the hourly quota results in slower service until traffic decreases. Exceeding the per-second limit results in a 429 error for the remainder of the second until the quota refreshes.
We could do a limiter based on this per client I suppose I guess but then I don't see why the end-user would not just use the current one and set the limit per hour and limit per second. We could create a rate limiter silently that enforces these per second and per hour limits if you wish
Thinking back based on what you say above you would say it would be nice for limits to be based per client. So we could implement the time-based limits per client (passing a client id or something). This makes other clients that are not throttled still free to make requests
Hey. It has been a while since this open source project was created and the API has changed since then. Blizzard already changed some of their API couple of years ago and we updated the repo in response. I am not sure if they have changed it again.
The idea of a rate limiter per client is that a user can provide us with many different keys and the rate limiter will cycle between the keys according to the limit set per key. This way you can exceed the limit provided by Blizzard for high traffic applications or needs.
Unfortunately, no one is currently maintaining this repo, so you will have to double-check that everything still works.
Thanks, and let me know if you have further questions.