confluentinc
Configuring max.poll.interval.ms above 12 hours results in an error
Environment Information
- OS [e.g. Mac, Arch, Windows 10]: linux
- Node Version [e.g. 8.2.1]: 18.x
- NPM Version [e.g. 5.4.2]: nil
- C++ Toolchain [e.g. Visual Studio, llvm, g++]: nil
- confluent-kafka-javascript version [e.g. 2.3.3]: 0.5.1
Steps to Reproduce
Configure max.poll.interval.ms to >12 hours using the KafkaJS consumer
confluent-kafka-javascript Configuration Settings
{
"max.poll.interval.ms": 86400000
}
Additional context
Results in this error:
Configuration property \"max.poll.interval.ms\" value 172800000 is outside allowed range 1..86400000
In CKJS the max poll interval that is passed is the double of what's configured. Because of the cache it has to maintain.
Given in CKJS the user isn't calling poll directly, to make sure that value corresponds to the max processing time of a single message or batch, first after max.poll.interval.ms all workers are awaited, if they don't complete processing during a second round of max.poll.interval.ms the consumer will lose the assigned partitions, otherwise the cache will be emptied and a new poll will be issued, preventing to reach max.poll.interval.ms
To configure 24h, the max value, you have to set 43200000, that will also correspond to the max processing time of a single message or batch in the worst case.
It would be good if the CKJS layer making this configuration transform also reported an error, rather than lowering a know-incorrect config to rdkafka which reports a nonsense error that requires the user to read the source code of CKJS to understand.
In 0.1.16, it was fine to configure the 24h maximum documented for rdkafka, and I was surprised by the error I received from 0.5.1 which made it seem like I changed the config and left me confused once I checked my config store and saw no changes.
Yep, agreed, I'll change the error message to reflect the internal doubling, the current error message is not helpful at all.
