propulsion icon indicating copy to clipboard operation
propulsion copied to clipboard

Add support for Backing off at Stream Level

Open bartelink opened this issue 2 years ago • 1 comments

Follow-on from #126 adding support for backoffs. Such backoffs might be useful for the following scenarios:

  • if a watchdog is monitoring a stream, it can have a quick look at any time, but it ca't and shouldn't do anything useful until the timeout has been reached. In this case, it can pass an Outcome of BackOffUntil time and the HandleOk can set it (except this PR does not pass the stream name to the success handler)
  • In general, the Cosmos SDK's intrinsic backoffs work ok and add some backpressure (at the cost of occupying a thread), which avoids the next handler invocation immediately also hitting the same issue. The mechanism here of backing off on one stream is as likely to be counterproductive (generating and sending another failing request right away) as productive (switch to another stream on another node that's not yet hit the rate limit threshold this time)
  • others? I'd like to see real scenarios so the design can actually be validated. While this code is low risk and relatively efficient, I really don't want to make people read it and/or deal with breaking changes and/or maintaining it unless it's actually proven to solve a legitimate need in the best way possible

bartelink avatar Nov 30 '21 11:11 bartelink

@ragiano215 This is what popped out - it has a breaking change and I am not personally using it yet so I might hold off on merging

bartelink avatar Nov 30 '21 11:11 bartelink