Group coordinator timeout

Open maeglindeveloper opened this issue 3 years ago • 7 comments

Description

Hi @edenhill,

I'm actually facing some issues regarding timeout with confluent-kafka-go 1.5.2. See below the logs:

Error 1

%4|1614175442.389|SESSTMOUT|rdkafka#consumer-27| [thrd:main]: Consumer group session timed out (in join-state started) after 6000 ms without a successful response from the group coordinator (broker 1, last error was Success): revoking assignment and rejoining group

Also having this

Error 2

%5|1614175531.192|REQTMOUT|rdkafka#consumer-31| [thrd:xxxxxx]: xxxxx/1: Timed out FindCoordinatorRequest in flight (after 60675ms, timeout #0)

Error 3

%5|1614177358.713|REQTMOUT|rdkafka#consumer-31| [thrd:xxxxxxxx]: xxxxx/1: Timed out ApiVersionRequest in flight (after 10010ms, timeout #0)
%4|1614177358.713|REQTMOUT|rdkafka#consumer-31| [thrd:xxxxxxxx]: xxxxxx/1: Timed out 1 in-flight, 0 retry-queued, 0 out-queue, 0 partially-sent requests

They are all related to timeout.

FYI: My microservice is initializing consumers to subscribe to 22 topics, each of them having 6 partitions, which makes one consumer per partition, each of them are belonging to the same group.id.

I'm running this microservice through telepresence.

I saw that there is a similar case mentionned in the confluent-kafka-python repository. https://github.com/confluentinc/confluent-kafka-python/issues/1011#issuecomment-767570642 Could it be related to the 1.6.0 version of librdkafka ?

Thanks in advance :+1:

Checklist

Please provide the following information:

[ ] confluent-kafka-go and librdkafka version (1.5.2)

Feb 24 '21 14:02 maeglindeveloper

@edenhill any advice ?

Mar 10 '21 15:03 maeglindeveloper

We'll release confluent-kafka-go v1.6.1 shortly, try to reproduce with that version (or current master branch) and if it still an issue reproduce with "debug": "cgrp,broker" to give us an idea what is happening.

Mar 11 '21 10:03 edenhill

getting this error after adding 5 brokers with replication factor 3

%4|1623843447.348|SESSTMOUT|EXCH_OMS_rslinmnmeftlotvpha_1#consumer-27| [thrd:main]: Consumer group session timed out (in join-state steady) after 30000 ms without a successful response from the group coordinator (broker 1, last error was Success): revoking assignment and rejoining group %4|1623843448.177|SESSTMOUT|AE_RES_rslinmnmeftlotvpha_1#consumer-13| [thrd:main]: Consumer group session timed out (in join-state steady) after 30000 ms without a successful response from the group coordinator (broker 5, last error was Success): revoking assignment and rejoining group %4|1623843449.305|SESSTMOUT|NSE_FOrslinmnmeftlotvpha_1#consumer-11| [thrd:main]: Consumer group session timed out (in join-state steady) after 30001 ms without a successful response from the group coordinator (broker 4, last error was Success): revoking assignment and rejoining group %4|1623843449.305|SESSTMOUT|NSE_FOrslinmnmeftlotvpha_2#consumer-2| [thrd:main]: Consumer group session timed out (in join-state steady) after 30002 ms without a successful response from the group coordinator (broker 4, last error was Success): revoking assignment and rejoining group %4|1623843449.306|SESSTMOUT|NSE_FOrslinmnmeftlotvpha_3#consumer-12| [thrd:main]: Consumer group session timed out (in join-state steady) after 30002 ms without a successful response from the group coordinator (broker 4, last error was Success): revoking assignment and rejoining group %4|1623843453.927|SESSTMOUT|BSE_FOrslinmnmeftlotvpha_1#consumer-5| [thrd:main]: Consumer group session timed out (in join-state steady) after 30001 ms without a successful response from the group coordinator (broker 4, last error was Success): revoking assignment and rejoining group %4|1623843453.928|SESSTMOUT|BSE_FOrslinmnmeftlotvpha_3#consumer-1| [thrd:main]: Consumer group session timed out (in join-state steady) after 30001 ms without a successful response from the group coordinator (broker 4, last error was Success): revoking assignment and rejoining group %4|1623843453.929|SESSTMOUT|BSE_FOrslinmnmeftlotvpha_2#consumer-22| [thrd:main]: Consumer group session timed out (in join-state steady) after 30002 ms without a successful response from the group coordinator (broker 4, last error was Success): revoking assignment and rejoining group