How to disable auto.commit

[yordis]

I have been trying to figure out how to disable auto.commit, I searched the docs and the code and tried to figure it out, but I am unfamiliar with the subject.

I would appreciate it if the documentation mentions such a setup since it is an important one we are trying to disable to ensure we do not lose messages.

Thanks in advanced.

[zmstone]

Hi @yordis do you mean you want to control when to commit offset?

if you use the v2 group subscriber, you can make the handle message callback return ack instead of commit. https://github.com/kafka4beam/brod/blob/master/src/brod_group_subscriber_worker.erl#L79 Then call the async commit API when it's ready to commit: https://github.com/kafka4beam/brod/blob/master/src/brod_group_subscriber_v2.erl#L191-L194

[yordis]

do you mean you want to control when to commit offset?

Something around the topic,

From https://docs.confluent.io/platform/current/clients/consumer.html#offset-management

By default, the consumer is configured to auto-commit offsets. Using auto-commit gives you “at least once” delivery: Kafka guarantees that no messages will be missed, but duplicates are possible. Auto-commit basically works as a cron with a period set through the auto.commit.interval.ms configuration property.

That may cause data loss in case of some processing issue, so I am trying to figure out how to disable the auto.commit all together

[zmstone]

do you mean you want to control when to commit offset?

Something around the topic,

From https://docs.confluent.io/platform/current/clients/consumer.html#offset-management

By default, the consumer is configured to auto-commit offsets. Using auto-commit gives you “at least once” delivery: Kafka guarantees that no messages will be missed, but duplicates are possible. Auto-commit basically works as a cron with a period set through the auto.commit.interval.ms configuration property.

That may cause data loss in case of some processing issue, so I am trying to figure out how to disable the auto.commit all together

That looks like a Java client behavior, and I am not entirely sure if it actually means it may cause data loss. brod never does this.