dataprep
dataprep copied to clipboard
Exponential backoff for 429 Error
Is your feature request related to a problem? Please describe. The connector should detect the HTTP code for the response. In the case of 429 too many requests error, it should wait for some time and retry the request.
Describe the solution you'd like
- Wait for an
asyncio.Event
object before each request. - Detect the response code if it is 429 then reset the event object, spin up a new future which will set the event with given delay.
Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.
Additional context
Take a look at https://docs.python.org/3/library/asyncio-sync.html#event to understand how it works. Use the backoff lib https://pypi.org/project/backoff/ for backoff algorithm.
Hi @kla55 I added some todos in feat/429-tolerence branch. I think you can take a look at the todos as the first step, and think about how to implement them.