kafka-rest icon indicating copy to clipboard operation
kafka-rest copied to clipboard

Set consumer setting max.poll.interval.ms to rest proxy setting of co…

Open mmercedes opened this issue 6 years ago • 7 comments

…nsumer.timeout.ms

If consumer.timeout.ms has been set to a value greater than the default value of max.poll.interval.ms and a consumer has set auto.commit.enable=false then it is possible the kafka brokers will consider a consumer as failed and release its partition assignments, while the rest proxy maintains a consumer instance handle. This then leads to an exception on the next call to poll, commitSync, or similar.

This commit sets max.poll.interval.ms equal to consumer.timeout.ms to ensure that kafka will not consider the consumer failed until the rest proxy does as well.

Steps to reproduce the issue :

  • Kafka version 0.10.1.0 or higher (where max.poll.interval.ms is introduced)
  • Set consumer.timeout.ms to a value greater than the max.poll.interval.ms default of 5 minutes
  • Create a consumer via the rest proxy
  • Sleep for a time greater than max.poll.interval.ms but less than consumer.timeout.ms
  • Have consumer attempt to read a record or post an offset

mmercedes avatar Dec 03 '18 19:12 mmercedes

It looks like @mmercedes hasn't signed our Contributor License Agreement, yet.

The purpose of a CLA is to ensure that the guardian of a project's outputs has the necessary ownership or grants of rights over all contributions to allow them to distribute under the chosen licence. Wikipedia

You can read and sign our full Contributor License Agreement here.

Once you've signed reply with [clabot:check] to prove it.

Appreciation of efforts,

clabot

ghost avatar Dec 03 '18 19:12 ghost

[clabot:check]

mmercedes avatar Dec 04 '18 16:12 mmercedes

@confluentinc It looks like @mmercedes just signed our Contributor License Agreement. :+1:

Always at your service,

clabot

ghost avatar Dec 04 '18 16:12 ghost

anyone able to take a look at this?

mmercedes avatar Feb 02 '19 21:02 mmercedes

Bump

bgreenlee avatar Mar 15 '19 20:03 bgreenlee

@stanislavkozlovski yes I have tested the issue locally and it resolved the issue I mentioned in my original comment. You can run these steps to reproduce the issue:

Steps to reproduce the issue :

  • Kafka version 0.10.1.0 or higher (where max.poll.interval.ms is introduced)
  • Set consumer.timeout.ms to a value greater than the max.poll.interval.ms default of 5 minutes
  • Create a consumer via the rest proxy
  • Sleep for a time greater than max.poll.interval.ms but less than consumer.timeout.ms
  • Have consumer attempt to read a record or post an offset

mmercedes avatar Mar 27 '19 15:03 mmercedes

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

cla-assistant[bot] avatar Sep 12 '24 17:09 cla-assistant[bot]