kombu
kombu copied to clipboard
celery
Connect to a random host in a multi-broker config
Add a feature to Kombu that allows it to connect to a random host in a list of brokers instead of always using the first node. This allows kombu to distribute connections to a cluster of brokers and minimize overloading any given broker.
Thanks!
Looks good, but there is already a way to do this in that you can set failover_strategy='shufflecycle'!
That uses shuffle, but you could also implement what you have here by doing:
Connection(failover_strategy=random.choice)
@ginoledesma This is a nice PR!
It's a bit unfortunate that the functionality was already in kombu. Do you mind if the PR is closed?
If you're still willing to help or want some fun problems to solve, there's still plenty of issues you could pick up. :smiley_cat:
Gladly! This is PR can be closed and I'll see what issues I can help with.
On Saturday, July 18, 2015, Tom White [email protected] wrote:
@ginoledesma https://github.com/ginoledesma This is a nice PR! [image: Thanks] https://camo.githubusercontent.com/9f37f319dbddf2652a28711bd9efe75c6327cc5b/687474703a2f2f6d656469612e67697068792e636f6d2f6d656469612f62736d463369583857724d6e432f67697068792e676966
It's a bit unfortunate that the functionality was already in kombu. Do you mind if the PR is closed?
If you're still willing to help or want some fun problems to solve, there's still plenty of issues https://github.com/celery/kombu/issues you could pick up. [image: :smiley_cat:]
— Reply to this email directly or view it on GitHub https://github.com/celery/kombu/pull/437#issuecomment-122584659.
For everyone else who is confused I just wanted to comment that setting the failover_strategy is not enough.
The first connection will always be opened to the first host in the list because a new host is only picked from the cycle when retrying a connection (calling maybe_switch_next()).
This PR could be merged as-is to fix the issue but unfortunately this wouldn't propagate to Celery users which are plagued by the problem of all workers connecting to the same broker. The only solution for now seems to be to shuffle the list of brokers after defining it.
I'm +1 on this, I just had a bad day from a bunch of identically configured nodes thundering connections at the same rabbit node. I'm tempted to patch in a random.shuffle(alt) right after https://github.com/celery/kombu/blob/master/kombu/connection.py#L174
@jason-kane As a workaround you could implement round-robin of all your hosts at the DNS level, depending on your DNS server it should return them in random order.