pyairtable
pyairtable copied to clipboard
gtalarico
Introduce RequestStrategy pattern
[Implements #142]
This PR introduces a new pattern for consumers to inject alternate request functionality into pyAirtable's ApiAbstract base class. A RequestStrategy should accept the request parameters pyAirtable uses, make the HTTP request to the Airtable API and then process the response and return the contained records in the format that pyAirtable expects.
Three implementations are included, SimpleRequestStrategy, RetryingRequestStrategy, and RateLimitRetryingRequestStrategy:
SimpleRequestStrategy encapsulates the use of a requests.Session to make the
requests, and expects the caller to provide any needed parameters like timeout
or headers. This behavior should be a 1:1 replacement of the old behavior.
SimpleRequestStrategy accepts an optional pre-configured Session if the caller
needs to do more specific set-up, e.g. of headers, etc.
RetryingRequestStrategy relies on tenacity.Retrying, from tenacity, a
well-established, widely trusted, and batteries-included function-retry library.
The caller must provide a configured Retrying object to control the retry
behavior on the requests.
RateLimitRetryingRequestStrategy subclasses RetryingRequestStrategy and provides a Retrying object that reimplements the built-in retry behavior of the official Airtable JS library, airtable.js. Requests to Airtable that hit the rate limit of 5 request/second, are rejected with a 429 status code, as are all subsequent requests for 30 seconds. airtable.js accommodates this with a jittered exponential backoff starting at 30s when it receives a 429, and that behavior is implemented here.
To Do
- [x] Add tests
- [x] Better type checking on request() to assert expected return shapes
@gtalarico I think the only thing still missing is the docs. I'm not really familiar with rst, but I could take a stab at it I suppose? The main instruction that needs documentation is to instantiate tables like so when you want the default retrying behavior:
from pyairtable import Table
from pyairtable.request_strategies import RateLimitRetryingRequestStrategy
API_KEY = "<api key>"
BASE_ID = "<base id>"
TABLE_NAME = "Table%20Name"
Table(API_KEY, BASE_ID, TABLE_NAME, request_strategy=RateLimitRetryingRequestStrategy)
Other than that, I believe the only other thing I haven't done is add integration test for the retrying logic, but that would require me to violate the rate limits, which seems like a bad idea...
Fwiw, I'm planning to start using this from my fork in some in-development code for now, so the code will get a full workthrough
@gtalarico This is ready to review when you have time... If you prefer / think it would be helpful I'm happy to schedule a video chat sometime for us to go over the PR together.
For docs, maybe a new section under api.rst - example below.
Then run make docs to preview.
Request Strategies
*******************
[ Overview, explain default behavior where strategy is None ]
[ Simple Example ]
Custom Request Strategy
--------------------------
[ Example ]
@gtalarico This is ready to review when you have time... If you prefer / think it would be helpful I'm happy to schedule a video chat sometime for us to go over the PR together.
Happy to as well. What timezone are you on?
Hi @gtalarico this week was a bit of a hell week for us, but I'm hoping next week is better... Let me get through the weekend and I'll respond to your comments and ping you to find a time that will work for us to go over this stuff
I was about to use pyairtable but I am sure I will also hit the rate limits issue. I am looking forward to seeing this PR merged. Until then, I will be using rest api. :)
Thank you for the great work, both of you!
Hi there, I'm going through and trying to clean up old PRs in this repo. Now that there is an option to pass a retry parameter, this issue seems resolved, so I'm going to close this pull request. If there's additional functionality you'd like to see that's not in main, please feel free to submit another patch.
