kcat
kcat copied to clipboard
edenhill
Feature idea: Allow plain JSON payload
Idea for a feature I'd find useful:
When I have JSON payload in Kafka and use kafkacat with the -J argument, I would like to have plain JSON as payload. Currently, the payload is always JSON-string-encoded, which makes it hard to use JSON parsers (like jq).
Managed to do it like this: kafkacat -C -t ... -e -J -u | jq '.payload=(if .payload==null then null else (.payload|fromjson) end).
I like this idea, question is what it should when the payload is not proper json, and thus might mess up the output:
- output as-is, this is the easy and performant option, but will break the message structure guarantee.
- verify that the json is correct first by deserializing, which is slow.
I would personally opt for alternative 1, since my applications already guarantee (and rely on) proper JSON. What I would find really useful would be a way to consume from a topic, change the key and/or value (e.g. with jq), and then produce the record to the same or a different topic. For that, we'd need a json producer as well. For my data it would be super useful.
