Unable to reset kafka offset intermittently, error "The consumer group must have no running instances, current state: Stable"

[shahtajk7]

Describe the bug I'm calling

await admin.setOffsets({groupId: gID, topic, partitions: await admin.fetchTopicOffsetsByTimestamp(topic, 1635358089189) });

before stating kafka consumer, but I ends up getting this error : The consumer group must have no running instances, current state: Stable sometimes, not always but almost on every alternate run.

To Reproduce

1. First letting kafka consumer to consumes from the latest offset.
2. terminating the the process via ctrl + c, it calls below method (Testing abrupt termination)

await consumerGroup.stop();
await consumerGroup.stop()
process.kill(process.pid)
process.exit(0)

3.I immediately re-starts my server. 4. On this run, the kafka admin will first reset the offset via above code, and then it'll connect the consumer to consume the messages. 5. Some times after re-starting the server, it works fine, offset gets reset and consumer runs perfectly, but on every other run when I re-start the server I'm getting this error : The consumer group must have no running instances, current state: Stable

Expected behavior When I'm calling consumerGroup.stop() and consumerGroup.stop() along with killing the process, then there should not by any running instance of consumer group when the server re-starts

Observed behavior Intermittently there is still any consumer group in running state even after stopping and disconnecting the consumer, due to which I'm unable to reset the offset, because we can't reset the offset if there is any running consumer instances. Reference link: The consumer group must have no running instances when performing the reset https://kafka.js.org/docs/admin#a-name-reset-offsets-by-timestamp-a-reset-consumer-group-offsets-by-timestamp

Environment:

• Mac OS 11.2.1
• KafkaJS version : 1.15.0
• Kafka version : 2.8.1
• NodeJS version : v13.8.0

Additional context Am I missing anything here? I'm a begginer in node.js and kafka and might be missing something obvious here so please point it out, or feel free to suggest any other solutions to fix this issue, Thanks.

[fberrez]

having the same issue with admin.resetOffsets({ groupId, topic });

[shahtajk7]

@fberrez I fixed this issue by using default sessionTimeout while creating consumer. Earlier we increased the timeout for some other reason, due to that consumer was keeping that group alive for that time period. Reverting it to default fixed the issue for me. But there should be some kind of other way to handle this properly.

[fberrez]

@shahtajk7 Thank you for your feedback. The sessionTimeout has not been changed before so its value is still the default one. I tried to play with it but it didn't change anything for my issue.

[Mellywins]

I'm running into this issue too using setOffset

[justinhaubrich]

SO am i

[daniel-kehlibarov]

Hi, can you stop the whole consumer group (several consumers) from one place for the same reason as everyone else here? I'm only able to stop the current consumer. Where does this come from and what does it achieve? Did you name your Kafka consumer consumerGroup?

await consumerGroup.stop();

Thanks in advance!

[tyrrellgit]

Have a look here, it helped me: https://medium.com/@curtis.porter/graceful-termination-of-kafkajs-client-processes-b05dd185759d

it might be that you need to add some error handlers to catch things and disconnect the consumers. Otherwise you have to wait for the timeout, from what I understand.