Add support for Kafka "raw" format for the payload

[gavin-shr]

The existing "format" configuration for the Kafka output plugin will put into the Kafka payload part a message in JSON format, MSGPACK format, or GELF format. In the JSON and MSGPACK formats this sends all the variables in the internal format, includes creating timestamp variable. This means that the clients which get the payload from the broker need to use these formats to extract the data, and also need to know the variables encoded in the data.

This patch adds an option for a "raw" format (configured using the Kafka output plugin "Format" configuration variable). This is to allow for any other format of the Kafka payload, which can be formed by either an input or filter plugin. For example, this could be used for sending a payload in Google Protobuf format. When using format "raw" by default, it will look for a variable named "payload" in the fluent-bit internal MSGPACK packet information, and send the contents as the Kafka payload. However, the configuration variable "Raw_Key" can be used to give the name of the variable containing the Kafka payload information (instead of using name "payload").

[edsiper]

thanks for your contribution.

would you please rebase on top of GIT master so we can test it ?

[edsiper]

ping

[t0rr3sp3dr0]

Any updates on this PR?

[vitalvi]

Hey, I came across this PR and I find it quite useful for my use case but it seems to be a bit forgotten. Can we have it merged?

[clotha]

This PR needs some attention. Not being able to send raw protobuf to kafka is the main reason we cannot switch to fluentbit now.

[niedbalski]

@gavin-shr Are you still interested to land this patch? If so, please rebase with git master and update conflicts.

Thanks!