kafka-connect-transform-common

kafka-connect-transform-common copied to clipboard

Hello,

We are pulling some data using kafka-connect (2.5.0) out of a kafka topic which contains JSON formatted data produced by winlogbeat. We only want the value from a specific key which we initially tried doing using ExtractField but it doesn't appear to support nested key/values. We stumbled upon this plugin which looks very promising but we're getting "MAP is not supported" errors.

Within our kafka-connect configuration, we are using the org.apache.kafka.connect.json.JsonConverter for both the keys and values. We have tried changing the over to the StringConverters to see if there would be any better luck and unfortunately not. The error message changes to "String is not supported".

We've tried chaining a mix of transforms from this plugin, a mix of this plugin and the default transforms (e.g. Flatten, Rename, Whitelist, ExtractField) that come with kafka-connect...every scenario seems to produce some type of error related to the type where it's expecting something specific and can't work with what was passed down.

I'm open to other transform chaining ideas/patterns but I'm kinda reaching out to see if MAP could be supported directly as I think our use-case is simple enough where others could benefit as well.