Limitations of an in-memory rate limiter

[iffies]

With the uploadAttachment API endpoint (quietly) introduced by Airtable in July 2024, ExAirtable's in-memory rate limiter is no longer suitable for large data processing without pre-throttling. You can include a 5MB file as a base64-encoded string in the data payload of a URL request. I'm using Oban to generate PDFs before sending them to Airtable over the uploadAttachment API. But even small ones at 20KB are blowing up memory usage in the rate limiter. The solution may be a chain of queues, with the file data kept in Postgres.

One of the niceties of Oban is that the Oban Web UI is now free, so you can track progress of your queues, see what failed, why it failed, what was cancelled, etc. So I guess it makes sense to add a secondary queue for slowing down requests to ExAirtable's rate limiter. Which begs the question of whether ExAirtable's rate limiter should become disk-based for dealing with attachments, and if so why not just switch to another solution like Oban. Then again, ExAirtable's rate limiter is rock solid in keeping requests under 5 per second.

And I don't think one wants to make ExAirtable dependent on Oban, even if Oban is free. And while Oban can limit job processing to five requests per second, I'm not sure if that is a paid Oban Pro feature, even on non-distributed deployments. And if you're not doing a lot of image processing in Airtable, you're not going to hit the rate limiter limitations. So maybe it's just a caveat on use of uploadAttachment that if one is sending queuing up thousands of requests, then you need to keep an eye on your memory usage.

I haven't added my PR yet for uploadAttachment, as I've raised some of the concerns with extending Airtable in another issue. With uploadAttachment, the format of the base URL has changed: from https://api.airtable.com/v0 to https://content.airtable.com/v0; and the table ID is no longer included in the URL because the Record ID uniquely identifies it. And, as with upserts, it needs a major rethink of whether one wants different data structures to capture all the fields of the requests and responses.

TLDR: uploadAttachment is a great new API and particularly well-suited to Elixir data orchestration, but it does show the limitations of ExAirtable's rate limiter since one will likely have to pre-throttle requests to it using an external job queue. It also requires complicating ExAirtable's current design that relies on a single data structure for responses and requests, and a fixed URL format. Since the beauty of ExAirtable is in its simplicity, perhaps it makes sense to create a new library that handles the new APIs introduced since late 2022.