internetarchive
Impossible to merge works - web client should respect rate limits
Problem
As increasingly aggressive rate limits have been enforced the web client hasn't been updated to respect those rate limits, making certain resource intensive operations, like the merge works screen rendering, impossible, even though they only represent a single user action.
This work merge:
https://openlibrary.org/works/merge?records=OL19883941W,OL18180666W,OL34839204W,OL31196671W,OL18964368W,OL19169319W,OL19025821W,OL18196302W,OL18674587W,OL18968678W,OL19479057W,OL18563718W,OL19600006W,OL19355339W,OL29309676W,OL25045090W,OL34963883W
causes an alert popup saying "Click OK to retry" which, of course, is exactly the wrong thing to do since it'll only dig the hole deeper.
Reproducing the bug
- Go to ...
- Do ...
-
Expected behavior: Web client backs off when it gets a rate limit exceeded error and retries the operation at the specified future time. If this is impossible for some reason, at least stop, warn the user, and let them manually retry the operation.
-
Actual behavior: Page rendering fails over and over without any indication to the user as to whats happening.
Context
- Browser (Chrome, Safari, Firefox, etc):
- OS (Windows, Mac, etc):
- Logged in (Y/N): Y
- Environment (prod, dev, local): prod
Breakdown
Requirements Checklist
- [ ]
Related files
Stakeholders
Instructions for Contributors
- Please run these commands to ensure your repository is up to date before creating a new branch to work on this issue and each time after pushing code to Github, because the pre-commit bot may add commits to your PRs upstream.
This merge still fails for me. There doesn't seem to be any handling of the 429 response in the web client (although it also doesn't look like any Retry-After or other hints are being returned either)
https://openlibrary.org/works/merge?records=OL8165007W,OL40238737W,OL41668313W,OL40423432W,OL30620069W,OL8367359W,OL40692889W,OL41135652W,OL40535436W,OL41712811W,OL41738078W,OL30660386W,OL41247566W,OL41401280W,OL40956554W,OL10009799W,OL40000586W,OL9878621W
Hmmm, looks like I don't have permission to reopen my own issue? I guess I have to create a new one.
Anecdotally, for n=1, preview for a merge of 18 works (not the one above) failed, but succeeded when trimmed to 14 works.
As an aside, it's a SUPER resource intensive page, with a whole bunch of stuff like lists, ratings, etc triggering individual API calls to return data that I absolutely never look at.
Here is another example from @onnotasler https://openlibrary.org/works/merge?records=OL8648038W,OL9105749W,OL9105919W,OL9105920W,OL9371828W,OL11081184W,OL28033815W,OL40406416W,OL40441124W,OL40577863W,OL41110324W,OL41371995W,OL10334313W,OL10334348W,OL10334374W,OL28161680W,OL28583192W,OL29449166W,OL29562064W,OL30734956W,OL30808225W,OL34810521W,OL30905124W,OL34902369W,OL34940018W,OL31083707W,OL35264366W,OL35264367W,OL31914235W,OL35763028W,OL36122717W,OL34804882W,OL37488490W,OL40239099W,OL40345017W,OL10437303W,OL26419966W&primary=OL26419966W&mrid=216387
I count 34 works in that URL, which is way higher than I'd even attempt. It looks like the preview page will render if it's trimmed to 24: https://openlibrary.org/works/merge?records=OL8648038W,OL9105749W,OL9105919W,OL9105920W,OL9371828W,OL11081184W,OL28033815W,OL40406416W,OL40441124W,OL40577863W,OL41110324W,OL41371995W,OL10334313W,OL10334348W,OL10334374W,OL28161680W,OL28583192W,OL29449166W,OL29562064W,OL30734956W,OL30808225W,OL34810521W,OL30905124W but I usually limit things to a dozen or so just to be on the safe side.
One usual trick that I've found is that you can merge a batch of works and then, WITHOUT clearing the selection, add another batch of works, merge those, and so on. The merged works only cost a single API call to find out that they're now a redirect, so are almost free from an API rate limit point of view. This approach saves having to start from scratch.
This merge request includes only 21 works, but I still run into the error: https://openlibrary.org/works/merge?records=OL502806W,OL502879W,OL502880W,OL502925W,OL502926W,OL12205611W,OL13097045W,OL15096991W,OL16325099W,OL16877299W,OL17861696W,OL17862572W,OL17863250W,OL24763508W,OL27045092W,OL33171182W,OL37388431W,OL42305647W,OL42322269W,OL42697474W,OL43656755W&primary=OL502806W&mrid=218253
Anecdotally, earlier in the week I was able to merge two different sets of 20 and then two different sets of 25, but the following day something much smaller, that I would have expected to work, 14?, failed, but then worked when I retried it, so I'm not really sure what they upper limit is and what factors influence it.
I think @cdrini offered a great suggestion during today's call which was to use a different set of rate limits for logged in v. logged out patrons.
Let's use 250 or 200 (more than the 180 we currently set) if patron is logged in. https://github.com/internetarchive/openlibrary/blame/63246c6ca2a808e3e33699a35453f979cb6a3e8c/docker/nginx.conf#L73
