Consumers unable to join group until group leader is restarted

[schmigware]

Description

Library version github.com/confluentinc/confluent-kafka-go/v2 v2.0.2 Kakfa broker version: 2.8.2 (Commit:3146c6ff4a24cc24)

Any input on the following would be greatly appreciated,

I have various consumers in consumer group FOO. The consumers are running on a number of k8s clusters and connecting to a common broker.

The consumers subscribe to various topics BAR0, BAR1, BAR2. Each of these topics has 32 partitions.

The observed behaviour is as follows:

• Kafka CLI tooling does not reflect ANY consumer as a consumer group member for topic BAR0.
• Consumer subscribes to topic BAR0 without error and is assigned an empty partition list.
• Kafka CLI tooling does not reflect the consumer as a consumer group member for topic BAR0.
• Kafka CLI tooling does not reflect ANY consumer as a consumer group member for topic BAR0.

This is the basic issue. Partitions don't appear to be assigned to any consumer.

Can be resolved by:

• Identifying the consumer group leader.
• Killing it.
• "Stuck" consumer is now assigned partitions.

This does not appear to be an issue with partition assignment within a group. Each consumer is subscribed to an unrelated topic. We are namespacing our Kafka topics to match them 1:1 to k8s clusters and namespaces. Certainly, the "stuck" consumer and the consumer group leader are not subscribing to the same topic.

No errors or warnings are observed in broker / consumer group leader logs.

How to reproduce

Not clear to us how to to reproduce this issue.