kafka-connect-elasticsearch
kafka-connect-elasticsearch copied to clipboard
Sink record timestamp as version number
Hi! Sometimes it's useful to take timestamp from kafka records as version number instead partition offset. For example when you add partitions in a target topic.
I agree, it's useful also if you need to make ES totally independent from KAFKA topics: by using the partition offset KAFKA acts as a primary storage. For example, on a sink configuration, if you empty/delete a source topic, you'll have also to empty/recreate the destination ES index. I had to patch the connector in order to use the record timestamp: it seems to work well on my scenario (time series storage)