kafka-rest icon indicating copy to clipboard operation
kafka-rest copied to clipboard

Set consumer setting max.poll.interval.ms to rest proxy setting of co…

Open mmercedes opened this issue 6 years ago • 7 comments

…nsumer.timeout.ms

If consumer.timeout.ms has been set to a value greater than the default value of max.poll.interval.ms and a consumer has set auto.commit.enable=false then it is possible the kafka brokers will consider a consumer as failed and release its partition assignments, while the rest proxy maintains a consumer instance handle. This then leads to an exception on the next call to poll, commitSync, or similar.

This commit sets max.poll.interval.ms equal to consumer.timeout.ms to ensure that kafka will not consider the consumer failed until the rest proxy does as well.

Steps to reproduce the issue :

  • Kafka version 0.10.1.0 or higher (where max.poll.interval.ms is introduced)
  • Set consumer.timeout.ms to a value greater than the max.poll.interval.ms default of 5 minutes
  • Create a consumer via the rest proxy
  • Sleep for a time greater than max.poll.interval.ms but less than consumer.timeout.ms
  • Have consumer attempt to read a record or post an offset

mmercedes avatar Dec 03 '18 19:12 mmercedes