A Consumer with no partitions assigned will timeout and leave the consumer group

[bobh66]

Describe the bug

If a Consumer has no partitions assigned, it will eventually timeout and leave the consumer group, which means that it will not be available for assignment on future rebalances, and must be restarted to rejoin the group.

This code:

https://github.com/aio-libs/aiokafka/blob/5d9af4d3dee17f9a9cffdbd904823cf61e54b442/aiokafka/consumer/group_coordinator.py#L750

triggers the consumer to leave the group when the fetcher is idle for max_poll_seconds, but when there are no partitions assigned the fetcher will always be idle.

Expected behaviour

A Consumer with no partitions assigned should remain in the Consumer group and be available for assignment on future rebalances.

Environment (please complete the following information):

• aiokafka version: 0.7.0
• kafka-python version: 3.8
• Kafka Broker version (kafka-topics.sh --version): 4.0.0-5.3.1
• Other information (Confluent Cloud version, etc.):

Reproducible example Bring up two consumers on a topic that has only one partition.

Run the command:

kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group --members

to see that 2 members are listed in the group but one has no partitions assigned.

After roughly 10 minutes the consumer that was not assigned a partition will leave the group and will no longer be available for assignment if the first consumer should fail.

Run the same command again and there is only one member shown in the group, assigned to the one partition.

[carlosjcabello]

This behavior is impacting me as well.

@ods Could you have a look at this?

We might want to make a change similar to https://github.com/aio-libs/aiokafka/pull/766 in the _heartbeat_routine (as done by @bobh66 on https://github.com/bobh66/aiokafka/commit/c786fa67564f6026c3076360cd6b42236a58c463) if we want the consumer to remain in the group if it has no assigned partitions. As explained by the OP, this way, future rebalances will include it as well.