API throttle detection option

[rbeede]

Sometimes the remote API endpoint returns back with HTTP 429 if too many requests are made to quickly. This can happen with as few as 79 documents to export from a folder.

Implement detection and a forced pause and retry if HTTP 429 is returned. Reduce the number of API calls made per second to accommodate.

Otherwise add a command line option to restrict to one download/thread at a time.

Example from export.log of failed run:

[29-03-2022 08:05:16.916] DEBUG: Couldn't fetch /threads/redactedidhere/export/docx, received 429
[29-03-2022 08:05:16.941] DEBUG: Couldn't fetch /threads/redactedidhere/export/docx, received 429
[29-03-2022 08:05:16.959] DEBUG: Couldn't fetch /threads/redactedidhere/export/docx, received 429
[29-03-2022 08:05:16.961] DEBUG: Couldn't fetch /threads/redactedidhere/export/docx, received 429
[29-03-2022 08:05:16.974] DEBUG: Couldn't fetch /threads/redactedidhere/export/docx, received 429
[29-03-2022 08:05:16.976] DEBUG: Couldn't fetch /threads/redactedidhere/export/docx, received 429
[29-03-2022 08:05:17.011] DEBUG: Couldn't fetch /threads/redactedidhere/export/docx, received 429
[29-03-2022 08:05:17.020] DEBUG: Couldn't fetch /threads/redactedidhere/export/docx, received 429
[29-03-2022 08:05:17.064] DEBUG: Couldn't fetch /threads/redactedidhere/export/docx, received 429
[29-03-2022 08:05:17.076] DEBUG: Couldn't fetch /threads/redactedidhere/export/docx, received 429
[29-03-2022 08:05:17.141] DEBUG: Couldn't fetch /threads/redactedidhere/export/docx, received 429
[29-03-2022 08:05:17.213] DEBUG: Couldn't fetch /threads/redactedidhere/export/docx, received 429
[29-03-2022 08:05:17.234] DEBUG: Couldn't fetch /threads/redactedidhere/export/docx, received 429

[osterman]

I was able to export ~500 documents. It would stall every few minutes but then pick back up again - presumably after limits were exhausted. This was using the hosted version (quip-export.com).

[ibrahima]

For what it's worth, I'm running this locally checked out from Git, and I noticed that it sometimes stalls for a while before continuing, and I assumed that was due to rate limiting. I haven't fully dug through the code, but it does seem to have some handling of rate limits.

It might be that you just need to change this line to also look for HTTP status 429?

---                if(res.status === 503) {
+++                if(res.status === 503 || res.status === 429) {

Hmm... it might be stalled because there's a per-hour limit of 750 requests... that seems unfortunate if you have more than ~700 documents.

https://quip.com/dev/automation/documentation/current#section/Rate-Limits/Per-user-Rate-Limits

It also seems like the Admin API might have fewer limits, but it may be a bit more complicated to get an admin API token than a personal API token (not sure if a personal API token automatically gets access to the admin API if the user is an admin).

[sonnenkern]

Hi ibrahima, thank you for using my tool.

It is very strange, I've tried to export from my Quip account without any issue. In the account I have about 86 folders, 213 threads and about 20MB of information. I have about 301 HTTP-queries during export, total duration about 3 minutes. I've received many 503-responses, but any 429. The important thing, if you receive 503-response, Quip-API tells you when can you repeat the request (x-ratelimit-reset header). But I don't know whether 429-response gives me the same information in x-ratelimit-reset header. There are any documentation about different http response codes.

Sure, I could treat 429-response in the same way as 503 and wait for several fix seconds before repeating the response, but I am not sure if it works.

How many threads do you have in your Quip Account? Maybe can you run export in Web App (www.quip-export.com) and look in the Browser-Console how 429-response looks like (especially response http-headers)?

Alternatively you could clone the Quip-Export repo, add treating of 429-responses and looks if it works for you.

Another thing... Quip has per-hour limits. I am not sure if it is the right way to freeze all further requests for one hour. Export partial folders might be the right solution in this case (--folders CLI parameter).

[ibrahima]

Oh yeah I should clarify that it did work for me. I never got any 429 errors. I was just responding to the original poster who seemed to be concerned about rate limiting/429 errors. I think my account has a bit more documents (about 2000 IIRC) so we probably hit the hourly rate limit but it did still work wonderfully in the end.

I should add that I sometimes get this error message when I've been running the script for a while and try to start over, which I suspect is also due to rate limiting, but I just back off for a bit and try later when that happens.

Quip-Export v2.3.2
ERROR: Token is wrong or expired.