hyx icon indicating copy to clipboard operation
hyx copied to clipboard

Introduce hedging

Open roma-glushko opened this issue 1 year ago • 0 comments

Sometimes it's reasonable to dispatch a few requests at the same time to organize a race between them for the sake of improving latency (while having some resiliency added). This is call hedging.

Usecases:

  • You have several LLM providers and you want to get the next agent "move" as soon as possible to meet real-time latency requirements

Reference:

  • https://github.com/resilience4j/resilience4j/tree/master/resilience4j-hedge

roma-glushko avatar Nov 05 '23 10:11 roma-glushko