kafka-connect-elasticsearch icon indicating copy to clipboard operation
kafka-connect-elasticsearch copied to clipboard

Sink record timestamp as version number

Open patsak opened this issue 8 years ago • 1 comments

Hi! Sometimes it's useful to take timestamp from kafka records as version number instead partition offset. For example when you add partitions in a target topic.

patsak avatar Apr 13 '17 10:04 patsak

I agree, it's useful also if you need to make ES totally independent from KAFKA topics: by using the partition offset KAFKA acts as a primary storage. For example, on a sink configuration, if you empty/delete a source topic, you'll have also to empty/recreate the destination ES index. I had to patch the connector in order to use the record timestamp: it seems to work well on my scenario (time series storage)

davidepozza avatar Aug 31 '17 15:08 davidepozza