dataprep icon indicating copy to clipboard operation
dataprep copied to clipboard

Exponential backoff for 429 Error

Open dovahcrow opened this issue 4 years ago • 1 comments

Is your feature request related to a problem? Please describe. The connector should detect the HTTP code for the response. In the case of 429 too many requests error, it should wait for some time and retry the request.

Describe the solution you'd like

  • Wait for an asyncio.Event object before each request.
  • Detect the response code if it is 429 then reset the event object, spin up a new future which will set the event with given delay.

Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

Additional context

Take a look at https://docs.python.org/3/library/asyncio-sync.html#event to understand how it works. Use the backoff lib https://pypi.org/project/backoff/ for backoff algorithm.

dovahcrow avatar Aug 12 '20 20:08 dovahcrow

Hi @kla55 I added some todos in feat/429-tolerence branch. I think you can take a look at the todos as the first step, and think about how to implement them.

dovahcrow avatar Aug 25 '20 05:08 dovahcrow