aiokafka icon indicating copy to clipboard operation
aiokafka copied to clipboard

[QUESTION] Does aiokafka have an equivalent to `enable.auto.offset.store=false` and `store()`

Open ntextreme3 opened this issue 2 years ago • 0 comments

Calling getmany seems to suggest it updates consumed offsets at the same time.

There are warnings in various places around manual commits, and suggestions that committing after each message is slow. Ex, from: https://github.com/edenhill/librdkafka/wiki/FAQ#why-committing-each-message-is-slow

There are actually two layers of offset commit, first there is the offset store (offset_store(), store_offsets(), etc) that locally stores (in memory) the offset to be committed, then there's the actual commit which commits the stored offset to the broker. An alternative is thus to explicitly store() each message's offset after processing but then relying on the auto offset committer to actually commit the offsets for you. Use enable.auto.offset.store=false to disable automatic offset store (which is by default done prior to your processing).

Is there anywhere in aiokafka that allows you to individually control the offsets that are eligible to be committed, while still using auto commit, similar to suggested above in that librdkafka FAQ -- or is manual commit the only option?

ntextreme3 avatar Mar 11 '22 23:03 ntextreme3