hyx
hyx copied to clipboard
Introduce hedging
Sometimes it's reasonable to dispatch a few requests at the same time to organize a race between them for the sake of improving latency (while having some resiliency added). This is call hedging.
Usecases:
- You have several LLM providers and you want to get the next agent "move" as soon as possible to meet real-time latency requirements
Reference:
- https://github.com/resilience4j/resilience4j/tree/master/resilience4j-hedge