kafka-connect-cosmosdb icon indicating copy to clipboard operation
kafka-connect-cosmosdb copied to clipboard

Setting lease container token with Kafka offsets does not handle multiple tasks

Open sivamu opened this issue 3 years ago • 2 comments

Description

Suppose we have a source connector with multiple tasks. Each task will try to reset the continuation token on the lease container (if using useLatestOffset=False) and there will be duplicate entries as each task will see it's own changes. The same applies when using useLatestOffset=True, since each task will attempt to rewind the changes and read them again.

Expected Behavior

Multiple tasks should be able to rewind the continuation token only once so that there won't be any repeat processing/duplicates.

Steps To Reproduce:

  • Configure Source Connector with single task
  • Insert some documents in cosmosdb
  • Reconfigure Source Connector with 2 tasks
  • Insert more documents in cosmosdb
  • Observe message feed in kafka

sivamu avatar Feb 10 '21 20:02 sivamu