Missing Support for Rate Limit Handling

[esb]

I'm migrating a bunch of code from Xeroizer, but I've noticed that there's no support for handling rate limit exceptions.

As I understand it, the API endpoint issues HTTP 429 errors and for retryable errors, includes a header indicating how long to wait before retrying. I would have thought that this is part of the API specification, and support should therefore be included in the SDK. It's an obvious candidate for inclusion in the client HTTP handling routines and makes the whole rate limit thing fairly transparent for the developer. At the moment, you need to include this rate limit handling around every API call, which is pretty tedious, and not really a good design practice.

[SerKnight]

Thanks for opening this issue @esb. We actually were discussing this as a team today!

If you can add any additional context as to what you would like to see, or any code samples of the rate limiting solution you would like incorporated that would be helpful.

[SerKnight]

This is the Xeroizer implementation which I will dig into. Though if you can link the additional context or code samples, hoping we can get a conversation going with the ideal implementation for any other community memebers.

---

https://github.com/waynerobinson/xeroizer#xero-api-rate-limits

Xero API Rate Limits

The Xero API imposes the following limits on calls per organisation:

A limit of 60 API calls in any 60 second period A limit of 5000 API calls in any 24 hour period By default, the library will raise a Xeroizer::OAuth::RateLimitExceeded exception when one of these limits is exceeded.

If required, the library can handle these exceptions internally by sleeping for a configurable number of seconds and then repeating the last request. You can set this option when initializing an application:

# Sleep for 2 seconds every time the rate limit is exceeded.
client = Xeroizer::PublicApplication.new(YOUR_OAUTH_CONSUMER_KEY,
                                         YOUR_OAUTH_CONSUMER_SECRET,
                                         :rate_limit_sleep => 2)

[esb]

I'm currently testing a wrapper around the call_api function using a Ruby module prepend. I can capture exceptions here and should be able to check for rate limit exceptions and retry them. I'll post the code once I've finished testing.

[esb]

Here's a first pass at implementing a rate limiter for the minute limit.

module RateLimit
  def call_api(http_method, path, opts ={})
    retry_attempts = 0
    retry_delay = 5
    retry_max = 5

    begin
      super
    rescue XeroRuby::ApiError => e
      if e.code == 429
        # Handle Rate Limit Exceptions
        rate_limit_problem = e.response_headers['x-rate-limit-problem']
        retry_after = e.response_headers['retry-after'].to_i

        if rate_limit_problem == 'minute'
          if retry_after > 0
            retry_attempts += 1

            if retry_attempts < retry_max
              puts "Rate Limit exceeded, retrying in #{retry_after + retry_delay} seconds"
              sleep(retry_after + retry_delay)
              retry
            else
              raise 'Rate Limit retry attempts exceeded'
            end
          else
            raise
          end
        else
          raise
        end
      else
        raise
      end
    end
  end
end

class XeroRuby::ApiClient
  prepend RateLimit
end

[mfkp]

A more interesting one to solve will be the "Concurrent Limit: 5 calls in progress at one time". I just ran into this one tonight.

https://developer.xero.com/documentation/oauth2/limits

For now, I just have all put all my Xero API calls into background jobs, and I used this Sidekiq throttling gem to make sure a maximum of 5 concurrent Xero API jobs are running at once: https://github.com/sensortower/sidekiq-throttled

It's not a great solution, but I don't know if there will be a good way to solve a "concurrent connection" rate limit through the ruby xero client itself...

[ghiculescu]

It's not a great solution, but I don't know if there will be a good way to solve a "concurrent connection" rate limit through the ruby xero client itself...

I don't think you can. The best way to handle this is to retry when rate limited. Xeroizer does it well.

[SerKnight]

Thanks for contributions ya'll. I think the simplest and non breaking solution would be an optional configuration param just like Xeroizer's solution :rate_limit_sleep => 2

Just want to make sure this encompasses enough configuration for bulk of rate limit issues before scoping out development and timeline.

[mfkp]

I think the simplest and non breaking solution would be an optional configuration param just like Xeroizer's solution :rate_limit_sleep => 2

That seems reasonable to me. Although I know that the error response also includes the "retry-after" => "10" header for the time-based rate limits, so maybe it could automatically retry after the requisite waiting period instead of retrying every x seconds? (and you would have a configuration option like :rate_limit_auto_retry => true)

Having a super long sleep call might not be ideal either, just locking up that process until it's done sleeping.

To make it even more complicated, the "retry after x seconds" plan gets hairier in some cases, e.g. I send a contact update (let's say I update a mailing address), it gets rate limited and will keep retrying. Then I make a second address update, which will also go into a retry loop. If the second update ends up going through before the first update, then the correct data might not be saved (unless the Xero library is also sending timestamps with the original call and can sort out how to discard old updates).

Just some things to think about 😅

[ghiculescu]

@SerKnight sounds good. Would be great to have this as similar as xeroizer as possible, for those looking to cut over!

[raouldevil]

@SerKnight Will the any rate limit apply to a tenant or to the authenticating Xero user? If we limit per tenant I guess this would be similar the Xeroizer functionality, but if the limit is applied per Xero user it would have implications for the case where one has multiple tenants linked using a single Xero user.

[iainad]

Just thought in case any one comes across this while we're waiting for a solution I'd update some small changes to the code posted by @esb for the current version (3.3.0) of the xero-ruby library with a little bit of refactoring.

module XeroRateLimit
  RETRY_DELAY = 5
  RETRY_MAX = 5

  def call_api(http_method, path, api_client, opts = {})
    retry_attempts = 0

    begin
      super
    rescue XeroRuby::ApiError => e
      raise unless e.code == 429
      raise unless e.response_headers['x-rate-limit-problem'] == 'minute'
      
      retry_after = e.response_headers['retry-after'].to_i
      raise unless retry_after.positive?

      retry_attempts += 1
      raise 'Rate Limit retry attempts exceeded' if retry_attempts >= RETRY_MAX

      Rails.logger.info "XERO Rate Limit exceeded, retrying in #{retry_after + RETRY_DELAY} seconds"
      sleep retry_after + RETRY_DELAY
      retry
    end
  end
end

If you are using rails you can prepend this in an initializer like so:

Rails.application.reloader.to_prepare do
  class XeroRuby::ApiClient
    prepend XeroRateLimit
  end
end

[chrisedington]

Thanks @iainad and @esb.

@iainad where would you put the module code? And the other smaller block as in initializer in config/initializers/xero_retry.rb for example?

[iainad]

@chrisedington apologies I just found your question sorry! I assume you figured it out but you can pretty much put it anywhere as long as the namespace works for zetiwerk (see here for more info on module naming/locations/loading in rails)

For me I stick stuff like this in a directory app/services so in this case app/services/xero_rate_limit.rb