Partitions are not divided evenly among consumers in a Consumer Group

[Smriti-OSS]

Versions

Please specify real version numbers or git SHAs, not just "Latest" since that changes fairly regularly.

Sarama	Kafka	Go
1.34.1	2.8	1.16

Configuration

What configuration values are you using for Sarama and Kafka?

Sarama Configuration:
 all are default except
 	clientConfig.Consumer.MaxWaitTime = 750ms
	clientConfig.Consumer.Group.Session.Timeout = 30sec
	clientConfig.Consumer.Group.Heartbeat.Interval = 6sec
	clientConfig.Consumer.MaxProcessingTime = 500ms

Logs

When filing an issue please provide logs from Sarama and Kafka if at all possible. You can set sarama.Logger to a log.Logger to capture Sarama debug output.

logs: CLICK ME

Problem Description

Hi,

We have a setup where we are running 16 consumers for a consumer group consuming from a topic with 16 partitions. We have noticed that even after the rebalancing operation, partitions are not evenly divided among the different consumers and usually just one consumer is reading from all the 16 partitions.

Any pointers on what could be causing this?

[Smriti-OSS]

Hi,

Found the problem. If you pass duplicate topics in group.Consume(KafkaTopics) then it leads to an issue in Balance_Strategy.go and most of the time empty assignments are made to the consumers as this line is executed https://github.com/Shopify/sarama/blob/53e77a4dd7182e049493dbce531d0dc5dbc31567/balance_strategy.go#L32-L34.

Passing unique set of topics fixed the issue for us and we could see an even division of consumers for the consumer group.

[Smriti-OSS]

Issue could be tested by adding this testcase to balance_strategy_test.go:

{ name: "2 members, 1 topic, 4 partitions each", members: map[string][]string{"M1": {"T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1"}, "M2": {"T1", "T1", "T1", "T1", "T1", "T1", "T1", "T1"}}, topics: map[string][]int32{"T1": {0, 1, 2, 3, 4, 5, 6, 7}}, expected: BalanceStrategyPlan{ "M1": map[string][]int32{"T1": {0, 1, 2, 3}}, "M2": map[string][]int32{"T1": {4, 5, 6, 7}}, }, },

This fails with:

=== RUN TestBalanceStrategyRange === RUN TestBalanceStrategyRange/2_members,_1_topic,_4_partitions_each#01 balance_strategy_test.go:101: Plan does not match expectation expected: sarama.BalanceStrategyPlan{"M1":map[string][]int32{"T1":[]int32{0, 1, 2, 3}}, "M2":map[string][]int32{"T1":[]int32{4, 5, 6, 7}}} actual: sarama.BalanceStrategyPlan{"M1":map[string][]int32{"T1":[]int32{0, 1, 2, 3, 4, 5, 6, 7}}} --- FAIL: TestBalanceStrategyRange (0.00s) --- FAIL: TestBalanceStrategyRange/2_members,_1_topic,_4_partitions_each#01 (0.00s)

[Smriti-OSS]

Fixed in: https://github.com/Shopify/sarama/pull/2285

[dnwe]

@Smriti-OSS i think we still want to de-dupe at the consumer level as well, like in your PR

[github-actions[bot]]

Thank you for taking the time to raise this issue. However, it has not had any activity on it in the past 90 days and will be closed in 30 days if no updates occur. Please check if the main branch has already resolved the issue since it was raised. If you believe the issue is still valid and you would like input from the maintainers then please comment to ask for it to be reviewed.

[dnwe]

Fixed in: #2285

Closing as fixed