probing pcap - wrong timestamps

[pelekhs]

Hello,

I have a use case where I need to extract Netflows or IPFIX from already collected traffic and not live traffic. I want to apply Nprobe on the traffic so as to exclude any payloads (for anonymisation purposes) but also apply the DPI of Nprobe and get layer 7 protocol identification. I have tried this locally running something like:

nprobe -I ./bigFlows.pcap -P . -T="%SAMPLING_INTERVAL %IN_BYTES %IN_PKTS %IPV4_SRC_ADDR %IPV4_DST_ADDR %IPV4_NEXT_HOP %IPV6_SRC_ADDR %IPV6_DST_ADDR %IPV6_NEXT_HOP %L4_SRC_PORT %L4_DST_PORT %SRC_VLAN %DOT1Q_SRC_VLAN %SRC_TOS %TCP_FLAGS %PROTOCOL %IP_PROTOCOL_VERSION %DIRECTION %FLOW_START_MILLISECONDS %FLOW_END_MILLISECONDS %INPUT_SNMP %OUTPUT_SNMP %IN_SRC_MAC %OUT_DST_MAC %ICMP_TYPE %BIFLOW_DIRECTION %L7_PROTO_NAME %HTTP_URL %HTTP_METHOD %HTTP_RET_CODE %DHCP_CLIENT_MAC %DHCP_CLIENT_IP %OUT_BYTES %OUT_PKTS %FIRST_SWITCHED %LAST_SWITCHED %FLOW_TO_APPLICATION_ID %FLOW_TO_USER_ID %INITIATOR_GW_IP_ADDR %EXPORTER_IPV4_ADDRESS"

In that way I try to save locally the exported netflows so as to safely publish them without payloads later on to another machine. However the timestamps I get with the templates %FLOW_START_MILLISECONDS %FLOW_END_MILLISECONDS are not the real timestamps that appear on the traffic. Instead they are probably the ones corresponding to when nprobe extracted the flows. How can I solve the problem? Should I use NTOPNG instead? As far as I am aware NTOPNG detects the real timestamps (First / Last Seen fields) however I haven’t been able to find a bulk way of exporting the flows from it right to the disk.

I also want to add that ideally after getting the netflows to another system I would like to get analytics on them using either ntopng or elasticsearch. Can you suggest the ideal steps in order to achieve this workflow?

Pcap saved in VM1 -> Nprobe/Ntopng -> Produce Netflows with real timestamps -> Migrate Netflows to VM2 -> Analytics with Ntopng/Elasticsearch in VM2.

Thanks in advance!