opensearch-connector-for-apache-kafka icon indicating copy to clipboard operation
opensearch-connector-for-apache-kafka copied to clipboard

Out of ordering of requests to opensearch

Open Ramkumar-byte opened this issue 6 months ago • 1 comments

Hi Team,

We are seeing an out of order problem for the requests to opensearch when using index.write.method=upsert.

However we don't see this issue when we set index.write.method=insert.

Below is our connector config

connector.class=io.aiven.kafka.connect.opensearch.OpensearchSinkConnector behavior.on.null.values=delete transforms.extractKey.field=id index.write.method=insert connection.password= tasks.max=10 topics=dev.task batch.size=10000 transforms=extractKey key.ignore=false max.in.flight.requests=5 key.ignore.id.strategy=none transforms.extractKey.type=org.apache.kafka.connect.transforms.ExtractField$Key schema.ignore=true key.converter.schemas.enable=false flush.timeout.ms=300000 value.converter.schemas.enable=false value.converter=org.apache.kafka.connect.json.JsonConverter connection.url=... key.converter=org.apache.kafka.connect.json.JsonConverter behavior.on.version.conflict=warn schemas.enable=false

Ramkumar-byte avatar Dec 22 '23 06:12 Ramkumar-byte

Hey @Ramkumar-byte, thanks for reporting this issue with the sink connector. Is it still happening for you?

qlecorre avatar Apr 25 '24 15:04 qlecorre