sof-elk
sof-elk copied to clipboard
use json output for nfdump data files
The JSON output mode for nfdump
should provide a better (faster?) and more streamlined processing pipeline than CSV. Some sample records are below. One potential optimization that would be useful would be an "ndjson" output mode, with one record per line and no enclosing []
list structure. This can be handled in post-processing, but it would be cleaner and probably more manageable for large data sets if it were native. I might submit a PR to nfdump
to add this.
[
{
"type" : "FLOW",
"sampled" : 0,
"export_sysid" : 1,
"t_first" : "2021-03-22T19:19:45.148",
"t_last" : "2021-03-22T19:20:17.420",
"proto" : 6,
"src4_addr" : "192.168.1.170",
"dst4_addr" : "192.168.3.161",
"src_port" : 49911,
"dst_port" : 9200,
"fwd_status" : 0,
"tcp_flags" : "........",
"src_tos" : 0,
"in_packets" : 3,
"in_bytes" : 204,
"connect_id" : "0",
"event_id" : "0",
"event" : "IGNORE",
"xevent_id" : "0",
"sgt_id" : "101",
"t_event" : "1970-01-01T00:00:00.0",
"label" : "<none>"
}
,
{
"type" : "FLOW",
"sampled" : 0,
"export_sysid" : 1,
"t_first" : "2021-03-22T18:35:56.12",
"t_last" : "2021-03-22T19:20:42.320",
"proto" : 6,
"src4_addr" : "192.168.1.170",
"dst4_addr" : "192.168.0.140",
"src_port" : 88,
"dst_port" : 378,
"fwd_status" : 0,
"tcp_flags" : "........",
"src_tos" : 0,
"in_packets" : 60911,
"in_bytes" : 5654215,
"connect_id" : "0",
"event_id" : "0",
"event" : "IGNORE",
"xevent_id" : "0",
"sgt_id" : "101",
"t_event" : "1970-01-01T00:00:00.0",
"label" : "<none>"
}
]
H/T to Raul P for the question that spurred this and the sample to prove it's viable.