backoff icon indicating copy to clipboard operation
backoff copied to clipboard

Allow retrying individual iterations of generator function's result

Open dmosorast opened this issue 2 years ago • 0 comments

When a function yields results from a failure prone process (i.e., a REST API pagination series), it would be ideal to bake the retry into that iteration in a clean fashion. This currently doesn't work with backoff, because the on_exception handler always uses return which breaks the yield chain and once the generator starts getting consumed, the backoff will not trigger.

I've come up with a minimal example to illustrate the point, and I think it could be fixed by checking if the return value is a generator and using yield from instead of return here.

It seems like retrying exceptions as the generator is iterating would be the preferred behavior and may be backwards compatible, but I don't think I'll be able to sit down and prove out all of the edge cases and get it tested properly, so I'm putting it here as a request for now 😄

Illustration (if you step into the call to backoffgen, you can see it skip the decorator on the iterations):

@backoff.on_exception(backoff.expo, Exception, on_backoff=lambda details: print("Backing off!"))
def backoffgen():
   for _ in range(1,10):
      if random.randint(0,10) < 5:
         raise Exception("BOOM!!!")
      yield 5
      time.sleep(1)

import ipdb; ipdb.set_trace()

for a in backoffgen():
   print(a)

dmosorast avatar Jun 23 '22 14:06 dmosorast