kafka-connect-elasticsearch
kafka-connect-elasticsearch copied to clipboard
Data Stream naming is far too restrictive
The approach to Data Stream naming in the connector is too restrictive. Right now the only option we have is to pattern the data stream name like so:
<data.stream.type>-<data.stream.dataset>-<topic>
This pattern doesn't even really allow us to match the common data steam naming scheme which, in kafka terms, would look a bit more like:
<type>-<topic>-<namespace>
And further, the data.stream.type can only be "metrics" or "logs".
None of these restrictions exist in elasticsearch itself; they are only imposed by the connector.
We should really be able to specify the entire data stream name, using some basic substitution variables (for topic name, etc.) in the string.
+1