kafka-connect-elastic-sink
kafka-connect-elastic-sink copied to clipboard
Push message to specific elastic index based on values of one or more fields of the record
I was thinking to propose a new configuration (like record.index.map) while setting up the connector where user can give a record field or combination of record fields which can be used to push this record to a particular ES index.
Eg: Record = {"fieldA" : "valueA", "fieldB" : "valueB", "fieldC" : "valueC", "fieldD" : "valueD"}
record.index.map = [fieldA]-[fieldC]
So, this record will be stored into ES index named: valueA-valueC As it will be a default configuration provided so any external Kafka connect plugin like RegexRouter and TimestampRouter may not be needed. And I am thinking this will be useful for many consumers. What's your thought on this @skynyrd ? Also, can you please add a license file to the project. It would be nice.