rabbitmq-server icon indicating copy to clipboard operation
rabbitmq-server copied to clipboard

khepri: `queue_leader_locator = balanced` doesn't behave as expected

Open mkuratczyk opened this issue 1 year ago • 0 comments

Note: this issue is related to an unreleased feature

When Khepri is enabled, queue_leader_locator = balanced doesn't work as expected (perhaps other strategies as well):

$ cat ~/balanced.conf
queue_leader_locator = balanced

$ bazel run start-cluster RABBITMQ_CONFIG_FILE=$HOME/balanced.conf

$ rabbitmqctl -n rabbit-0 enable_feature_flag raft_based_metadata_store_phase1

$ rabbitmqctl -n rabbit-0 import_definitions 5k-quorum-queues.json
Importing definitions in JSON from a file at "5k-quorum-queues.json" ...

# while the import is running, check how the leaders are distributed, for example:
$ rabbitmqctl --no-table-headers --quiet -n rabbit-0 list_queues leader | sort | uniq -c
    744 rabbit-0@kura

# at the end, with 5k queues, I have a skewed distribution:
$ rabbitmqctl --no-table-headers --quiet -n rabbit-0 list_queues leader | sort | uniq -c
   2327 rabbit-0@kura
   1301 rabbit-1@kura
   1372 rabbit-2@kura

The balanced strategy behaves differently for the first 1000 queues and then just distributes queues randomly. Judging by the above, that first 1000 just goes to the first node (instead of being precisely distributed) and then the rest is indeed distributed randomly.

The issue is likely somewhere in: https://github.com/rabbitmq/rabbitmq-server/blob/main/deps/rabbit/src/rabbit_queue_location.erl

mkuratczyk avatar Sep 01 '22 15:09 mkuratczyk