Support A "Hedge" LoadBalancer

[ryanjbaxter]

See this Spring Tip for the basic idea https://spring.io/blog/2019/01/23/spring-tips-hedging-client-requests-with-the-reactive-webclient-and-a-service-registry

Here is some code https://github.com/spring-tips/hedging/blob/52a40255435eccb711808b4779aad8d639cc2264/client/src/main/java/com/example/client/ClientApplication.java#L40-L100

[ryanjbaxter]

@spencergibb @OlgaMaciaszek would love your thoughts on an implementation. Here are some options I have been thinking of

1. We could add a separate annotation, but this seems a bit confusing since it is really just a change in the algorithm we are using
2. Add a property to @LoadBalancer something like @LoadBalancer(algorithm="roundrobin") this would be the default. Then we could have @LoadBalancer(algorithm="hedge"). This approach seems more flexible so if we have other algorithms in the future that would be easy. The confusion with this approach is that we already have a way to configure the algorithm via ribbon that is different than this.

[spencergibb]

/cc @joshlong

[joshlong]

Hi! I would love to see something like this in the SCC. There was a really good comment on the YouTube video for my Spring Tip by a Ashok Koyi, which i reproduce here, that bears consideration.

"I understand its just an example, but this implementation has various problems

1. It does not handle errors properly. If any of the upstream servers always fail fast due to any issue at its end, the overall client call will fail even if another service returns the response with slightly more latency
2. You shared a HashMap across multiple threads (most commons Scheduler implementation), which is completely unsafe
3. This does not work with all load balancer algorithms. Many people prefer latency based next server selection (and not round robin). If this high value algo is selected, this solution becomes completely useless as loadBalancer always returns the best server according to the past statistics (the same server again and again) & only single call to the past best server is always selected
4. If by the time between you retrieve service ids from discover client & the time you ask load balancer for next server, if the service registry info changes, say a bunch of servers leave due to network partition, your application does not behave the way you intend, i.e you might have less number of upstream servers than you intended (or) you may get no server returned if all he servers are invisible now

Overall, the next best server selection should be done inside the loadbalancer as he might have latency information. The concept is fine, but your implementation needs to be changed significantly to be anywhere near to be used in a production grade system due to the issues I mentioned above"

[spencergibb]

I think we could do this. I'd prefer to start in the spring-cloud-loadbalancer. There could be a specific implementation that you would enable via @LoadBalancerClient(configuration=MyConfig.class).