Produce message from a file with headers

[evgeniyasti]

Hello

I need to produce batch messages to Kafka so I have a file that I feed kafkacat:

 kafkacat -b localhost:9092 -t <my_topic> -T -P -l /tmp/msgs

The content of /tmp/msgs is as follows

-H "id=1"
{"key" : "value0"}
-H "id=2"
{"key" : "value1"}

When I run the kafkacat command above, it insert four messages to kafka - one message per line in /tmp/msgs.

I need to instruct kafkacat to parse the file correctly - that is -H "id=1" is the header for the message {"key" = "value0"}.

How do I achieve this?

Thanks

[edenhill]

There is currently no way to express per-message headers, the specified headers are used for all messages.

With the future json-based producer we'll allow headers per message.

[masoncj]

It would be wonderful if there was a way we could feed a JSON stream from kafkacat -C -J into kafkacat -P -J and have it produce exactly the same stream of messages. This would be incredibly useful for copying messages from one broker to another for instance.

[Tommmster]

@edenhill Is there any work being done (complete or not) on the json-based producer? I'm looking for the functionality mentioned in the above message (consuming with -P -J) and it's not reflected on the documentation or the source code (confirmed with a quick test on 1.7.0). Looked into getting started with an implementation, but thought i'd ask first. If not, any tips are welcome. Thanks!

[masoncj]

@Tommmster check out https://github.com/edenhill/kcat/pull/295