confluentinc
How is sessionTimeout being used?
In the docs for consumers, you have the rebalanceTimeout config:
The maximum allowed time for each member to join the group once a rebalance has begun. Note, that setting this value also changes the max poll interval. Message processing in eachMessage/eachBatch must not take more than this time.
To me this implies that setting this, overrides/is used in place of the sessionTimeout, especially given the docs make no mention of this property.
However, the KafkaJS ConsumerConfig type has a sessionTimeout value and it appears to be using the value when creating the config for rdkafka.
I'm just wondering what is the correct usage for this?
Relevant KafkaJS docs:
- https://kafka.js.org/docs/consuming#a-name-each-message-a-eachmessage
- https://kafka.js.org/docs/consuming#a-name-options-a-options
The rebalanceTimeout is client-sided, while the sessionTimeout is broker-sided.
If you set the rebalanceTimeout to something like 10s, it will timeout rebalances in 10s, and it also expects you to process each message or each batch within 10s. In case you aren't able to do that, the client library itself declares itself to be a failed consumer, and indicates to the broker that it can trigger a rebalance of the group. This isn't a permanent failure, if you continue processing then the "failed consumer" can rejoin the group.
The sessionTimeout is when, if your application isn't sending heartbeats for that much time, the broker will remove you from the consumer group. Because we have many threads in the application, you don't need to worry about doing this yourself, the library takes care of it in the background.
TL;DR, whatever you set rebalanceTimeout to, please process messages/batches quicker than that. You don't need to worry about sessionTimeout in most cases for your applications.
Understood. Thank you for the explanation, it's much appreciated! 🙏
You don't need to worry about sessionTimeout in most cases for your applications
Is there ever a scenario where configuring sessionTimeout would make sense for this library?