Version conflict when consuming records same same partition

[Harish-Sridhar]

Hi,

Recently i faced the version conflict issue when providing a key as id to index documents to Elastic search. It is because the offset of the consumed record is used as version.

It makes sense that version conflict arises when the records are produced over different partitions of a topic in a round robin fashion.

But what i don't understand is, version conflict arises, even when the messages are in same partition.

Am i missing something? Can some one clarify?

Regards, Harish.

[Harish-Sridhar]

Could it be because both the indexing operation was part of a single bulk request and the ordering is not guaranteed within a bulk request?

[sleminov-tc]

Having same issue. Documents ids are set via message keys such that all updates to same document are coming via same partition. Partitions count never changes.

ES: 7.11.2 ES sink connect: 11.0.3 Base connect: 6.1.1

1. Case where versions are same. Record is never get indexed.

io.confluent.connect.elasticsearch.ElasticsearchClient$ReportingException: Indexing failed: [postman_test.domain.0/QRM2UsdLSwO8xi4M1398CQ][[postman_test.domain.0][4]] ElasticsearchException[Elasticsearch exception [type=version_conflict_engine_exception, reason=[b3a8ac5b-7443-4164-b9aa-aacdfeaab789]: version conflict, current version [3] is higher or equal to the one provided [3]]]

2. Case where versions are different. Higher version is indexed.

io.confluent.connect.elasticsearch.ElasticsearchClient$ReportingException: Indexing failed: [opensrs.domain.0/F1ZLPT0USgmbWuFvzUDK_Q][[opensrs.domain.0][0]] ElasticsearchException[Elasticsearch exception [type=version_conflict_engine_exception, reason=[45a8ac5b-7443-4164-b9aa-aacdfeaab345]: version conflict, current version [50] is higher or equal to the one provided [49]]]

[senim2]

Any updates on this issue?