goflow
goflow copied to clipboard
add xtra flag to send json on Kafka transport
The protobuffer has support for marshaling protobuffer messages as json: google.golang.org/protobuf/encoding/protojson
For some popular tools, like logstash (that are more text oriented), having json would be a better option as message format. (See https://github.com/logstash-plugins/logstash-codec-protobuf/issues/15). At least it will make it a lot easier to share data between (legacy) systems.
Basically the impact would be minimum: in the Kafka transport the protobuffer message is marshaled as json format instead of wire format based on a flag. Fields of type bytes are then converted as base64 strings which is, for some systems, safer.
I've implemented a working version for this: https://github.com/gjelu/goflow/commit/e2a0c56de200a6dc99f69ab3561efd4c280b3b51
Some extra required changes:
- I am using protojson from the protobuf APIv2, so I need to convert the v1 message to a v2 message.
- for this I need to upgrade to github.com/golang/[email protected]
I can do a PR if it's feasible that this can be accepted? Or if you have any extra comments on this?
It would be very useful feature for me.
I'd love to see this feature!