How consumer group support multiple consumers consume the same group?

[yitian108]

Versions

Please specify real version numbers or git SHAs, not just "Latest" since that changes fairly regularly.

Sarama	Kafka	Go
1.37.0	2.5.0	1.17.11

Now, I try to use Shopify/sarama library(kafka go client) to implement multiple consumers in the same consumer group, and I have the following testing: Suppose the topic have 3 partitions, and the group Id is "groupId":

TEST-1

(1)

newClient, err := sarama.NewClient(brokerList, &cfg)

(2)

 group1, err := sarama.NewConsumerGroupFromClient(groupId, newClient)
 group2, err := sarama.NewConsumerGroupFromClient(groupId, newClient)
 group3, err := sarama.NewConsumerGroupFromClient(groupId, newClient)

The testing is try to use a kakfa client(newClient) to create multiple groups, and the program will block one group, and the other two groups will not response, it seems to block the following code:

group.Consume(ctx, strings.Split(topics, ","), consumer)

and then a moment later, the program rebalance (setup/clean up) repeatly, the result is that the consumer consumed all partitions, not each consumer consume per partition.

TEST-2

(1)

newClient1, err := sarama.NewClient(brokerList, &cfg)
group1, err := sarama.NewConsumerGroupFromClient(groupId, newClient1)

(2)

newClient2, err := sarama.NewClient(brokerList, &cfg)
group2, err := sarama.NewConsumerGroupFromClient(groupId, newClient2)

(3)

newClient3, err := sarama.NewClient(brokerList, &cfg)
group3, err := sarama.NewConsumerGroupFromClient(groupId, newClient3)

In this testing, we can see each consumer group with a new client, the result indicates that each consumer will consume a partition, which accord with expection.

So, How can I implement multiple consumer groups bind the same kafka client, rather than creating a new client per consumer group?

[yitian108]

Multiple kakfa consumer with the same groupId and the shared one kafka client @eapache @dnwe Could you please check this would be an issue?

[simon28082]

The partition feature of kafka doesn't support such behavior, but there is another idea to consume it and redistribute it downstream, similar to a proxy, which we are doing and it works well

[yitian108]

That sounds good. I believe we would look forward for this feature, and when will the new function be published?

[github-actions[bot]]

Thank you for taking the time to raise this issue. However, it has not had any activity on it in the past 90 days and will be closed in 30 days if no updates occur. Please check if the main branch has already resolved the issue since it was raised. If you believe the issue is still valid and you would like input from the maintainers then please comment to ask for it to be reviewed.

[Pkfire26]

Multiple kakfa consumer with the same groupId and the shared one kafka client @eapache @dnwe Could you please check this would be an issue?

Hi, I'm just wondering what the goal would be to have multiple consumer groups on the same client. I'm not sure what sharing a client would accomplish. Just inquiring for the sake of understanding the kafka better.

[github-actions[bot]]

Thank you for taking the time to raise this issue. However, it has not had any activity on it in the past 90 days and will be closed in 30 days if no updates occur. Please check if the main branch has already resolved the issue since it was raised. If you believe the issue is still valid and you would like input from the maintainers then please comment to ask for it to be reviewed.