netsampler
SAMPLING_INTERVAL not used when included in the data template
MikroTik devices do not include the SAMPLING_INTERVAL field in the Option Data Sets, instead it's included in the Data Set and defined by the Data Template - the problem is GoFlow2 never looks for or uses the SAMPLING_INTERVAL field in the Data Set, so the SamplingRate field in the output is always 0.
I am not sure if this is out of spec and the MikroTik doing things wrong, but to resolve the issue I had to tweak producer/producer_nf.go:
Add the following in ConvertNetFlowDataSet()
case netflow.NFV9_FIELD_SAMPLING_INTERVAL:
DecodeUNumber(v, &(flowMessage.SamplingRate))
Alter the line fmsg.SamplingRate = uint64(samplingRate) to prevent overwriting the above with 0:
if samplingRate > 0 {
fmsg.SamplingRate = uint64(samplingRate)
}
I haven't submitted the above as a pull request as I'm not 100% sure this is the best way to handle this scenario, alongside the existing sampling system logic. I have a pcap if useful.
Sidenote: MikroTik devices seem to send the sampling interval in little endian rather than big endian expected by DecodeUNumber(), so an interval of 256 is actually coming through as 65536 - I'm correcting that at a later stage.
Hi @washcroft, Thank you for the report, I was aware in the past Mikrotik users were setting a manual sampling rate but didn't know about this. Your fix should be sufficient.
For the second part, I may make sure the field exists in Option Data Template instead of checking for zero.
If possible, do you have a packet capture of a Data Template and Data Set to confirm the endianness and with other values (eg: 24576)? This definitely sounds strange. Could you also share the model/OS version?
Sure, tested this with Mikrotik CCR2004 RouterOS 7.4.
I did test some other values, here are the results in WireShark - happy to share full pcap files privately.
take 1 : skip 255 (should be 256 not 65536)
take 1 : skip 128 (should be 129 not 2164260864
take 32 : skip 128 (should be 5 not 83886080)
take 1 : skip 0 (packet sampling switched off) - the field is set to zero
Thank you. I am hesitant adding it into the producer. This should be supported using the custom mapping options. Would you be able to test https://github.com/netsampler/goflow2/pull/130 which supports little endian as part of the custom mapping?
You can find the binaries here.
I would also raise an issue to Mikrotik as this does not seem normal.
This should be supported using the custom mapping options.
I have had a quick look at your changes, unfortunatly controlling endianness in the mapping means you're applying an endianness "fix" on every flow exporter, not just the devices/exporters that need it.
Also, it seems the endianness mapping setting is then applied to every integer type field, in this case the Mikrotiks are already behaving as they should on all fields except SAMPLING_INTERVAL so this is likely to cause more issues.
Endianness aside, this is still an issue:
MikroTik devices do not include the SAMPLING_INTERVAL field in the Option Data Sets, instead it's included in the Data Set and defined by the Data Template - the problem is GoFlow2 never looks for or uses the SAMPLING_INTERVAL field in the Data Set, so the SamplingRate field in the output is always 0.
I have had a quick look at your changes, unfortunatly controlling endianness in the mapping means you're applying an endianness "fix" on every flow exporter, not just the devices/exporters that need it.
This is tricky. I want to avoid making filters to apply partial fixes. The aim of mapping is to support enterprise fields rather than fixing bugs. My suggestion would be to have another instance of GoFlow2 on a different port so the mapping can be applied.
Also, it seems the endianness mapping setting is then applied to every integer type field, in this case the Mikrotiks are already behaving as they should on all fields except SAMPLING_INTERVAL so this is likely to cause more issues.
That's not expected. It's possible I introduced a bug. Did you use the configuration provided?
My suggestion would be to have another instance of GoFlow2 on a different port so the mapping can be applied.
Fair enough.
That's not expected. It's possible I introduced a bug. Did you use the configuration provided?
It's not a bug you introduced - it's the main reason behind opening this issue. The bad endiannes Mikrotik's are using when sending their sampling interval was a side note and unrelated to the first/orignial post on this issue.
I believe this is resolved. See https://github.com/netsampler/goflow2/blob/app/refactor/docs/protocols.md#map-custom-fields (#150) also for mapping examples.
Feel free to re-open if I missed something