goflow icon indicating copy to clipboard operation
goflow copied to clipboard

add xtra flag to send json on Kafka transport

Open gjelu opened this issue 4 years ago • 2 comments

The protobuffer has support for marshaling protobuffer messages as json: google.golang.org/protobuf/encoding/protojson

For some popular tools, like logstash (that are more text oriented), having json would be a better option as message format. (See https://github.com/logstash-plugins/logstash-codec-protobuf/issues/15). At least it will make it a lot easier to share data between (legacy) systems.

Basically the impact would be minimum: in the Kafka transport the protobuffer message is marshaled as json format instead of wire format based on a flag. Fields of type bytes are then converted as base64 strings which is, for some systems, safer.

I've implemented a working version for this: https://github.com/gjelu/goflow/commit/e2a0c56de200a6dc99f69ab3561efd4c280b3b51

Some extra required changes:

  • I am using protojson from the protobuf APIv2, so I need to convert the v1 message to a v2 message.
  • for this I need to upgrade to github.com/golang/[email protected]

I can do a PR if it's feasible that this can be accepted? Or if you have any extra comments on this?

gjelu avatar May 25 '20 15:05 gjelu

It would be very useful feature for me.

Slepwin avatar Jan 15 '21 20:01 Slepwin

I'd love to see this feature!

mugugnu avatar Jan 29 '21 16:01 mugugnu