goflow2 icon indicating copy to clipboard operation
goflow2 copied to clipboard

Add UDP Transport

Open door7302 opened this issue 3 years ago • 1 comments

Sometime to transfer (without batching) flows exports from goflow2 to a third party app (Enricher...) which is not hosted on the same server/VM a UDP simple stream is needed. This new driver allows to push data generated by Goflow by using a UDP socket.

This new transport driver provides this config options : "transport.udp.dst": Udp remote IP destination "transport.udp.port": Udp remote port "transport.udp.src": Udp local source IP address to use "transport.udp.pack_pb": Use this option to pack several pb message per UDP pkts. fixlen option and pb encoding are required" "transport.udp.mtu": Specify the mtu to avoid IP fragmentation "transport.udp.num_socket": "Use this option to have more entropy (one source port per socket) - useful for client that relies on REUSEPORT socket option"

This driver has been tested at scale (more than 350K flows/s) whit pack_pb option set, and Jumbo UDP Mtu set to 9000 and num_socket = 50 to allow better kernel load balancing on the third party app side.

It would be very cool if this enhancement might be integrated into the main branch of Goflow2. David

door7302 avatar Jul 15 '22 15:07 door7302

Thank you for this PR. Still considering adding this feature as it's one more transport to maintain. At the same time, I would like to see if this generate interest from other users. In terms of similar designs, this reminds me of Juniper devices sending Telemetry protobuf over UDP (other option was gRPC). Didn't have a great experience with it though. Not super fond of UDP as it would be harder to implement message delivery guarantees.

lspgn avatar Jul 21 '22 05:07 lspgn

Coming back to this. It's also possible to use pipe and netcat (understanding this won't be able to use transport.udp.pack_pb).

goflow2 | nc localhost 5001

Additionally, in terms of use-cases, I realized syslog could be implemented with a new format and netcat too

https://github.com/golang/go/blob/f57f02fcd519a86683c47a444e0e24a086fea8e0/src/log/syslog/syslog.go#L292

lspgn avatar Mar 29 '23 05:03 lspgn