netsampler
How can a Format get access to custom fields in a ProtoProducerMessage?
Not sure where to ask this, so I filed an issue. Let me know if there is a better channel for questions.
I am trying to migrate netflow2ng to goflow2.
In order to do this, I am using goflow2 to as a library, and I wrote and registered a couple of new Formats (there's a custom JSON format and a another custom binary format), and then a ZMQ Transport. I have the basics of this working.
As part of doing this, I ran into Issue #186, where goflow2 will prefer OUT_BYTES/OUT_PACKETS for the values for the FlowMessage.Bytes/Packets Protobuf. To get around this, I followed the advice in that thread and used a Yaml config to map all four IN/OUT NFv9 fields to custom proto fields (901, 902, 903, 904).
My problem is that from inside my new Formats Format() function, I am passed an interface which is really a ProtoProducerMessage, which is a wrapper around a FlowMessage. I cannot figure out a way to access the custom proto fields (901, 902, 903, 904) in ProtoProducerMessage. I have verified they are there. If I use the built in Json or Text formatters, these fields are there. But looking at the goflow2 code, it tools like doing that relies on ProtoProducerMessage.mapUnknown(), which is not exported.
What's the right way to do this?
TL;DR: I want to write my own Formats that has access to both IN_BYTES and OUT_BYTES from NFv9. I don't think I want my own Producer, I'm happy with all the work the current NFv9 sampling producer is doing, I just need to keep IN_BYTES and OUT_BYTES. Actually I really only need to keep IN_BYTES, IN_PACKETS.
In netflow2ng, I ended up remapping 4 fields and compiling my own [.proto file] (https://github.com/synfinatic/netflow2ng/blob/main/proto/extended_flow.proto), it works fine, but I don't love that we'll have to recompile to stay in sync with goflow2 if this changes. Hopefully this doesn't happen often?
Alternatively I still think goflow2 should have InBytes/Packets and OutBytes/Packets in FlowMessage. But, doing that breaks backwards compatibility. I could do a PR for that if you want?