blackcandy spams discogs with requests on large media library

[maunzCache]

I happen to have a rather large media library and just wanted to have everything there on blackcandy. For my installation, i set up an API key for discogs to pull artist information and covers that are missing.

However, after ~10-15 artists the log shows failed attempts for getting more images. The log runs like crazy and i assume it is due to

retry_on Integrations::Service::TooManyRequests, wait: 1.minute, attempts: :unlimited

I have ~1600 albums there. When the first 60 requests come in there is 1540 more requesting. So waiting one minute is not enough wait time in my case. My suggestion is to pull information from discogs in batches or have at least an exponential back-off for the waiting time.

This might by like a rare case, but its happening.

Edit: Here is a note about rate limiting at discogs

Requests are throttled by the server by source IP to 60 per minute for authenticated requests, and 25 per minute for unauthenticated requests, with some exceptions.