Problem saving IPFIX data

[joerglad]

Hello, we are using nfdump v 1.6.22 for storing Netflow V9 data and it works fine. Now we try to store IPFIX data, generated by an Extreme VSP switch. The data is received and stored by nfcapd, the details are correct, but only the byte counter is always set to zero. (the IPFIX flows contain real values for the byte counter) Here is the output of a flow in raw format (nfdump -o raw):

Flow Record: Flags = 0x06 NETFLOW v10, Unsampled label = export sysid = 8 size = 60 first = 1660023678 [2022-08-09 07:41:18] last = 1660023678 [2022-08-09 07:41:18] msec_first = 532 msec_last = 674 src addr = 192.168.1.66 dst addr = 192.168.10.1 src port = 41622 dst port = 161 fwd status = 0 tcp flags = 0x00 ........ proto = 17 UDP (src)tos = 0 (in)packets = 11 (in)bytes = 0 ip router = 192.168.98.161

Here is the data template sent by the switch, output by wireshark:

What can be the reason for this? Thanks and Regards Jörg

[phaag]

Hi Jörg, Bytes total tag 85 has a byte length of 6. The ipfix decoder in 1.6.x can only handle well known var lengths of 2, 4, 8 bytes etc. therefore it ignores the 6 bytes announcement. Either you may try to convince your exporter to send either 4 or 8 byte counters or you may try the nfdump unicorn branch, which becomes 1.7 soon. nfdump 1.7.x has a more flexibel ipfix decoder in many ways. It also accepts those odd counter sizes.

[joerglad]

Hi Peter, thanks for the hint. I tried the unicorn, but with mixed results:

• IPFIX flows has now a byte count, that is correct in most cases
• but sometimes the byte counter has an impossible value (256 TB in 5 packets), i will try to capture the corresponding IPFIX-packets. Here is a sample flow: Flow Record: Flags = 0x00 NETFLOW v10, Unsampled Elements = 3: 1 2 12 size = 84 engine type = 0 engine ID = 0 export sysid = 13 first = 1660637100394 [2022-08-16 10:05:00.394] last = 1660637100613 [2022-08-16 10:05:00.613] received at = 1660637101638 [2022-08-16 10:05:01.638] proto = 17 UDP tcp flags = 0x00 ........ src port = 44255 dst port = 161 src tos = 0 in packets = 5 in bytes = 256194799206400 src addr = 192.168.7.201 dst addr = 192.168.10.2 ip exporter = 192.168.98.161

I have also another problem with nfdump (v1.7) filter expressions that worked in Version 1.6: the filter '( src net 172.16.17.0/24 or dst net 172.16.17.0/24 )' generate the error message Line 1: syntax error at '/24 or dst net 172.16.17.0/' Do you have an idea here?

Regards Jörg

[joerglad]

Hi Peter, I've captured a packet with the terabyte issue: this is the IPFIX packet from device: IP (tos 0xb8, ttl 251, id 0, offset 0, flags [none], proto UDP (17), length 126) xxx.xx.98.161.2055 > xxx.xx.1.66.9995: [udp sum ok] UDP, length 98

    0x0000:  45b8 0057 0000 0000 fb11 40a4 xxxx 62a1 
    0x0010:  xxxx 0142 0807 270b 0043 fba3 000a 003b 
    0x0020:  62fb a6cc 000f 6c26 0000 0001 0101 002b 
    0x0030:  xxxx 64fb c0a8 0a01 af1d 00a1 1100 0000 
    0x0040:  1300 0000 0010 ce00 0001 82a7 0389 e800 
    0x0050:  0001 82a7 038d 8c 

the byte count is 0000 0000 10ce, in decimal 4302 (this value is plausible for 19 packets in flow)

and this the corresponding flow record: Flow Record: RecordCount = 107099 Flags = 0x00 NETFLOW v10, Unsampled Elements = 3: 1 2 12
size = 84 engine type = 0 engine ID = 0 export sysid = 5 first = 1660659403240 [2022-08-16 16:16:43.240] last = 1660659404172 [2022-08-16 16:16:44.172] received at = 1660659404459 [2022-08-16 16:16:44.459] proto = 17 UDP tcp flags = 0x00 ........ src port = 44829 dst port = 161 src tos = 0 in packets = 19 in bytes = 226568114798592 src addr = xxx.xx.100.251 dst addr = 192.168.10.1 ip exporter = xxx.xx.98.161

Idea: Byte-count in IPFIX packet is 0000 0000 10ce Byte count saved in flow recort is 226568114798592 in Hex this is ce10 0000 0000 - reverse of original byte count may be there is a big endian / little endian problem?

Regards

Jörg

[phaag]

Do you see this on all records? That would be a systematic issue.

[phaag]

What System/CPU are you using?

[phaag]

Could you share the pcap? If so, please send it to the email in the AUTHORS file. The filter issue should be fixed in the latest unicorn push.

[joerglad]

The system is CentOS Linux release 7.9, running on Intel Xeon E5 CPU, 16G RAM. The problem only occours with a few IPFIX records, most byte counter are ok. The pcap is comming soon.

Thanks and regards Jörg

[joerglad]

The filter expressions are working correctly for me now.

[phaag]

After a few back and forth debugging cycles with @joerglad - many thanks for your support @joerglad , we came to the conclusion, that an old CentOS and a new compiler have issues, which show up in this bug. It turned out, that a variable is not yet fully written, but already used for next instructions. Therefore this issue will be closed.

[joerglad]

I have now tested nfdump on a system with CentOS 8 Stream and it works fine. All IPFIX records are saved correctly. I've used the latest source from github and gcc v.8.5, provided by the distribution. @phaag - Thanks for good support Jörg

[phaag]

Very welcome!