Is there any configuration property in source connector to notify the delete of es document. in Kafka topic.

[shanthkumar079]

We are able to insert data to sink(Postgres) using the Kafka-connect elasticsearch-source from a data source(elastic search). I have a use case to delete a document from elastic search and the same should be reflected in the data sink(Postgres).

[DarioBalinzo]

Hi @shanthkumar079, No, it is not possible to detect deletion in elasticsearch using this connector.

However, I suggest to add a new boolean field (e.g. deleted=true/false) to logically delete records in elastic and use that field in postgres to discard logically deleted data. In addition to this you may delete from postgres old data based on retention based.

Hope this helps, Dario

[shanthkumar079]

Hi @DarioBalinzo, Thanks for your response:-) I was trying to update a document(value for some fields) by the document Id, seems like this connector doesn't support the update as well. Please correct me, if my understanding is wrong.

Thanks Shantkumar.s

[DarioBalinzo]

The connector supports updates if you use an incremental timestamp field (e.g. last update time).