elixir-lang
Add fair dispatcher
Attempt to provide more even distribution to consumers than demand dispatcher. It should allow different max demands but prevent greedy consumers - unless other consumers are exhausted. Also allows splitting up the batches of events when demand from multiple consumers is larger than events dispatched.
We need to handle situations where a single dispatch currently sends multiple batches to the same consumer because the round robin does more than one loop.
Yup! And GenStage master passes the length to dispatch, so you can calculate how many batches goes to a given process.
RoundRobinDispatcher is marginally more characters and it is immediately clear to readers.
I know this is a quite old PR - but curious if there has been any more thought around this additional type of dispatcher?
I have a scenario that could greatly benefit from something like this where I use GenStage to distribute work to a large number of distributed commodity (spot instance) nodes/servers. It works fine, but depending on things like spot instance prices, I'd like the ability to bring on instances of various sizes/computing power and have them subscribe with demand numbers that reflect their inherent resources (cpu count, etc). I'm absolutely open to any criticisms around this idea, so if i'm thinking about this idea wrong, let me know.
Obviously I can roll my own dispatcher, but just wanted to dig this up from the grave before I go down that road :)
@fishcakez @josevalim
@amokan I believe you would need a weighted equivalent to this, otherwise it is likely that an instance with double the processing power of another may get the same number of events.
@fishcakez Thanks for the feedback and that does make sense. I'm going to look closer at your implementation on Monday and work through some test cases and see what I can come up with in regards to the idea around weight.
