goflow2

goflow2 copied to clipboard

Opening an issue to jot down a few ideas. I think a refactor is going to be necessary, not super satisfied how samples are processed anymore.

Use-cases have changed over the years:

• More towards JSON export, protobuf still strong for unifying diverse samples
• More fields necessary (Enterprise/custom)
• Requiring custom handling over the messages that aren't data (eg: Options templates in NetFlow/IPFIX)
• Less code edits when adding fields, library for decoding being used externally

The producer element is too strict I believe. Should think about using reflect and allow populating maps. The message dispatch could also be improved (sharing socket for NetFlow). Less buffer copy operations? Transport could be improved, especially with Kafka, by adding delivery guarantees and improving access to configuration options. At the same time, would like to avoid becoming a multi-output tool like Telegraf or Vector.

Other objectives on the roadmap

• Ensure MPLS labels are stored in an array #100
• Optimize performance using sync.Pool to reuse protobufs.
• Write encoder for testing
• Investigate if custom mapping fields can be added to a protobuf as unknown fields if do not exist

Check if possible: extend mapping to "create" at runtime new protobuf fields?

This commit is trying to change the protobuf to match the spec case.

Avro can be alternative to Protobuf for the next version. Here is a really fast Avro codec: github.com/hamba/avro

It's faster than protobuf. More efficient. It's row based, so it can be splitable later on. Kafka supports it. It supports Snappy compression. And also thanks to Avro Object Container Files, you don't need to add any schema on Kafka or Clickhouse etc.

Decoding must be less expensive than Protobuf btw. And output sizes should be smaller.

Investigate setting the default to reuseport --> removing one flag

@yusufozturk There are few serialization formats that are interesting like Cap'nProto, Avro, Thrift... Unfortunately to make the producer agnostic of the source structure, this would require quite an intensive rewrite. It's something part of a bigger project. In the current setup, the efficiency would be lost in conversions.

Closing as v2 is released