New soft goal TopicLeaderReplicaDistributionGoal?

[rhardouin]

Hi, We would like to evenly distribute topics leadership across brokers. A kind of mix between TopicReplicaDistributionGoal and LeaderReplicaDistributionGoal.

We plan to implement a custom soft goal e.g. TopicLeaderReplicaDistributionGoal. The idea would be to use it along with TopicReplicaDistributionGoal to mimic the behavior of Kafka assigner mode but without moving all replicas "from scratch".

Is it a good approach? Does it make sense for an upstream contribution or is there a better option? (e.g. modify an existing goal) Thanks

[efeg]

Hi @rhardouin Thanks for creating this issue! To better understand the motivation, could you tell a little bit about the scenarios that this goal intends to help with?

The #921 and corresponding PR would be relevant.

[rhardouin]

Hi @efeg

We've split our workload into multiple Kafka clusters (roughly one cluster per service instead of having one big cluster with non-spanning topics). In each cluster, each topic spans all brokers and partitions load is homogeneous. Thus replicas and leader replicas must be spread evenly across all brokers to ensure an homogeneous load.

Images speak louder than words so here is the situation on a cluster:

TopicReplicaDistributionGoal is able to spread evenly replicas across brokers

But leadership is imbalanced because LeaderDistributionGoal works at broker level:

Topic A handles more traffic than other topics so you can see that broker 13 with 2 leaders is idle compared to brokers having 7 or 8 leaders.

[3286360470]

I have the same needs and thoughts.It's a good idea, because most of the time the workload of each partition of a topic is closed to each other, but the workload between topics is different.

[efeg]

@rhardouin Thanks for the detailed explanation. The use-case seems reasonable, and creating a new goal rather than modifying an existing one is preferred.

We look forward to your upstream contribution with the new goal!

[amuraru]

Adding here another use-case for this:

the __consumer_offsets topic is one such example where the leaders should be evenly distributed across brokers ( when a consumer group is created in kafka the group coordinator is elected one of the leaders of this topic serving the randomly selected partition for that CG).

While "LeaderReplicaDistributionGoal" goal can be used to achieve this for all topics it is not easily configurable to target a specific set of topics.

As such a new soft goal should be added TopicLeaderDistributionGoal that requires a set of topics to be configured and balances evenly the leaders across all brokers

[3286360470]

I have the same needs and thoughts.It's a good idea, because most of the time the workload of each partition of a topic is closed to each other, but the workload between topics is different.

I have implemented TopicLeaderReplicaDistributionGoal locally, and the actual effect of the prd-cluster is also very good, Just poll the candidateBrokers(which sorted by comparing numLeaderReplicasOfTopicInBroker, then comparing the brokers' NW_OUT、NW_OUT、brokerId etg.)

[rhardouin]

Hey sorry for the delay, it's been a while since I pushed a draft on our fork , I still have to:

• make all tests green
• make the goal work correctly when combined with TopicReplicaDistributionGoal

Despite these rough edges we tested it on production clusters and it worked well... if not combined with TopicReplicaDistributionGoal. When we use both goals it seems they work against each other instead of converging to a good solution.

NOTE: I had to use a ConcurrentHashMap in Broker class to avoid a race condition. It could also occur with existing goal TopicReplicaDistributionGoal because the for loop over topics is the same

[amuraru]

@rhardouin Hi Romain - any progress on. this? I can help with testing if needed. thanks

[rhardouin]

@amuraru Unfortunately I wasn't able to put more time on it since my last commit

[rhardouin]

@amuraru Yes some help would be welcome :pray:

[bpux]

@rhardouin I have same needs and can you please let me know how I can test your goal? do I use only this goal to rebalance or I can combine all other goals(except TopicReplicaDistributionGoal I guess). thanks.

[rhardouin]

@bpux There is no silver bullet: sometimes we run the goal alone sometimes we can add another goal but usually we use multiple passes (we tend to do that without this new goal anyway). For instance a first iteration with DiskUsageDistributionGoal + TopicReplicaDistributionGoal + ... and once done we equalize leaders with TopicLeaderReplicaDistributionGoal. This step often requires to exlude other topics e.g. __.* or even some other business topics e.g. ([^t].+|specific_topic_name)

[jlei-nr]

Hello @efeg! I'm wondering if you could point me in the right direction. I have an implementation of this but as a hard goal that I was thinking about PR-ing. Should I open a new issue or would it be ok to piggy-back on this one?

[efeg]

Hi @jlei-nr, Please feel free to create a PR and link it to this issue.