nProbe icon indicating copy to clipboard operation
nProbe copied to clipboard

Flows not being ingested into ClickHouse DB

Open FooChuanTian opened this issue 2 years ago • 0 comments

Hi,

We are trying to use nProbe to capture IPFIX packets being sent to us from NetFlow exporters on separate machines and store them in a Clickhouse Database. However, we found that the flows are not being ingested into the DB fully. The following error is displayed:

: While executing ParallelParsingBlockInputFormat: data for INSERT was parsed from stdin: (in query: INSERT INTO flows (IN_SRC_MAC,OUT_DST_MAC,INPUT_SNMP,OUTPUT_SNMP,SRC_VLAN,IPV4_SRC_ADDR,IPV4_DST_ADDR,L4_SRC_PORT,L4_DST_PORT,IPV6_SRC_ADDR,IPV6_DST_ADDR,IP_PROTOCOL_VERSION,PROTOCOL,L7_PROTO,L7_CONFIDENCE,IN_BYTES,IN_PKTS,OUT_BYTES,OUT_PKTS,FIRST_SWITCHED,LAST_SWITCHED,CLIENT_TCP_FLAGS,SERVER_TCP_FLAGS,L7_PROTO_RISK,L7_RISK_SCORE,EXPORTER_IPV4_ADDRESS,DIRECTION,SAMPLING_INTERVAL,TOTAL_FLOWS_EXP,NPROBE_IPV4_ADDRESS,POST_NAT_SRC_IPV4_ADDR,POST_NAT_DST_IPV4_ADDR,POST_NAPT_SRC_TRANSPORT_PORT,POST_NAPT_DST_TRANSPORT_PORT) FORMAT CSV): (in file/uri (fd = 0)): (at row 581) . (CANNOT_PARSE_INPUT_ASSERTION_FAILED)

The following command was used for nProbe: nprobe --zmq "tcp://*:5556" -i none -n none --collector-port 2055 -T "@NTOPNG@" --clickhouse=127.0.0.1:ntop::default:(PASSWORD)

nProbe Version: 10.2.230720 Build OS: Ubuntu 22.04.2 LTS License: nProbe Enterprise L

Thank you.

FooChuanTian avatar Jul 25 '23 03:07 FooChuanTian