sof-elk icon indicating copy to clipboard operation
sof-elk copied to clipboard

use json output for nfdump data files

Open philhagen opened this issue 3 years ago • 1 comments

The JSON output mode for nfdump should provide a better (faster?) and more streamlined processing pipeline than CSV. Some sample records are below. One potential optimization that would be useful would be an "ndjson" output mode, with one record per line and no enclosing [] list structure. This can be handled in post-processing, but it would be cleaner and probably more manageable for large data sets if it were native. I might submit a PR to nfdump to add this.

[
{
    "type" : "FLOW",
    "sampled" : 0,
    "export_sysid" : 1,
    "t_first" : "2021-03-22T19:19:45.148",
    "t_last" : "2021-03-22T19:20:17.420",
    "proto" : 6,
    "src4_addr" : "192.168.1.170",
    "dst4_addr" : "192.168.3.161",
    "src_port" : 49911,
    "dst_port" : 9200,
    "fwd_status" : 0,
    "tcp_flags" : "........",
    "src_tos" : 0,
    "in_packets" : 3,
    "in_bytes" : 204,
    "connect_id" : "0",
    "event_id" : "0",
    "event" : "IGNORE",
    "xevent_id" : "0",
    "sgt_id" : "101",
    "t_event" : "1970-01-01T00:00:00.0",
    "label" : "<none>"
}
,
{
    "type" : "FLOW",
    "sampled" : 0,
    "export_sysid" : 1,
    "t_first" : "2021-03-22T18:35:56.12",
    "t_last" : "2021-03-22T19:20:42.320",
    "proto" : 6,
    "src4_addr" : "192.168.1.170",
    "dst4_addr" : "192.168.0.140",
    "src_port" : 88,
    "dst_port" : 378,
    "fwd_status" : 0,
    "tcp_flags" : "........",
    "src_tos" : 0,
    "in_packets" : 60911,
    "in_bytes" : 5654215,
    "connect_id" : "0",
    "event_id" : "0",
    "event" : "IGNORE",
    "xevent_id" : "0",
    "sgt_id" : "101",
    "t_event" : "1970-01-01T00:00:00.0",
    "label" : "<none>"
}
]

philhagen avatar Mar 26 '21 12:03 philhagen

H/T to Raul P for the question that spurred this and the sample to prove it's viable.

philhagen avatar Mar 26 '21 12:03 philhagen