Beam Kafka Consumer not committing offset to the configured consumer group

[rhaces]

When running a pipeline with a Beam Kafka consumer, the kettle-beam job will not commit the offset to kafka and therefore if the process is restarted it will duplicate messages (using auto.offset.reset set to earliest)

This has been tested with Neo4j as the output of the job.

[mattcasters]

As usual for parallel environments, the answer is complicated...

https://stackoverflow.com/questions/48406321/kafkaio-checkpoint-how-to-commit-offsets-to-kafka

[mattcasters]

I made the option "Allow offsets to be committed back" the default.