utoni
Conf file
Hi Toni, sorry for posting here a request...
we would like to propose a new feature with the nDPId, a special argument for a .conf file instead of having all options on the command line. we could have something like that:
- command: ./nDPId -x /etc/nDPId/nDPId.conf
- (-x or -C or whatever you think it should be fine with you)
- nDPId.conf file including options: interface=eth0 logfile=/var/log/nDPId/ndpid.log socket=/tmp/ndpid-collector.sock or 127.0.0.1:7000 background=yes protofile=/etc/nDPId/myprotos.txt etc...
it could also be nice to have a special compilation option (ex: -DCONF_FILE=/etc/nDPId/nDPId.conf)
do you think it makes sense to you ?
have a nice day, Fred
I had this already in my mind. But I thought nobody would be interested in having this feature. Will do ASAP.
Toni, it will be readable and useful :o) then all popular products use that way with settings...
not an emmergency, just a "nice-to-have".
bye Fred
Toni,
giving a try, we keep you posted once compiled and started...
bye Fred
Thanks! Hoping to get some feedback from you guys! (positive and negative) :smile_cat:
compiled, launched, it seems all is ok........................... for the moment. -R and -f work fine.
we keep it running in our lab for a while, we keep you posted. anyway, thanks a lot for your work !!!
bye Fred
Hi Toni,
we have detected a strange behaviour with the conf file, let me explain...
we have used the 'ndpid.conf.example' (renamed ndpid.conf) you provide with the original source but those 3 options seems to act not correctly, we did not use them with previous builds: #internal = true #external = true analysis = true
we kept 'internal' and 'external' commented so it means not enabled in the settings. but the option 'analysis' seems not working properly, actualy the values (from the json) of 'flow_id' and 'flow_first_seen' are incoherent, indicating many MB up to many GB with protos (ex: ICMP, DNS, ...). we have not those strange values with the command line, surely because those options are 'false' by default.
by setting those 3 options to false we now get normal values.
so our advise is to modify the file 'ndpid.conf.example' and set 'false' by default with those 3 options or with a warning that 'people must take blablabla...' especialy with the 'internal'.
except that point, latest build works fine. :o)
bye Fred
by "actualy the values (from the json) of 'flow_id' and 'flow_first_seen' are incoherent", we mean values are always the same and sizes are increasing with incoherent value...
Toni, here is the settings working fine with us:
[general] ... internal=false external=false compression=true analysis=false
[tuning] max-flows-per-thread=2048 max-idle-flows-per-thread=64 max-reader-threads=10 daemon-status-interval=600000000 #memory-profiling-log-interval=5 compression-scan-interval=20000000 compression-flow-inactivity=30000000 flow-scan-interval=10000000 generic-max-idle-time=600000000 icmp-max-idle-time=120000000 tcp-max-idle-time=180000000 udp-max-idle-time=7440000000 tcp-max-post-end-flow-time=120000000 max-packets-per-flow-to-send=15 max-packets-per-flow-to-process=32 max-packets-per-flow-to-analyse=32 error-event-threshold-n=16 error-event-threshold-time=10000000
[ndpi] packets_limit_per_flow=32 flow.direction_detection=enable flow.track_payload=disable tcp_ack_payload_heuristic=disable fully_encrypted_heuristic=enable libgcrypt.init=1 dpi.compute_entropy=1 fpc=enable # you set it 'disable' dpi.guess_on_giveup=0x03 flow_risk_lists.load=1 flow_risk.crawler_bot.list.load=1 log.level=0
[protos] tls.certificate_expiration_threshold=30 # we set it back to default tls.application_blocks_tracking=disable # we set it back to default stun.max_packets_extra_dissection=25 # we set it back to default
bye Fred
Hi Fred
we have used the 'ndpid.conf.example' (renamed ndpid.conf) you provide with the original source but those 3 options seems to act not correctly, we did not use them with previous builds: #internal = true #external = true analysis = true
we kept 'internal' and 'external' commented so it means not enabled in the settings. but the option 'analysis' seems not working properly, actualy the values (from the json) of 'flow_id' and 'flow_first_seen' are incoherent, indicating many MB up to many GB with protos (ex: ICMP, DNS, ...). we have not those strange values with the command line, surely because those options are 'false' by default.
If the analysis mode is enabled, the daemon will consume much more heap memory. The reason behind that is that it needs additional memory necessary to store some statistical values until max-packets-per-flow-to-analyse reached before firing an analyse event which contains all those statistical values. The heap memory used for that is free'd afterwards.
by "actualy the values (from the json) of 'flow_id' and 'flow_first_seen' are incoherent", we mean values are always the same and sizes are increasing with incoherent value...
That's something I do not understand. Could you please elaborate this more (or provide an example)?
But I'll comment out analysis = true, which will disable the analysis completely and add some comments about memory consumption.
by "actualy the values (from the json) of 'flow_id' and 'flow_first_seen' are incoherent", we mean values are always the same and sizes are increasing with incoherent value...
That's something I do not understand. Could you please elaborate this more (or provide an example)?
Ya I know, not easy to understand, we spent hours to see the issue and tried to identify where the problem could be. let me summarize what we saw:
- using your .conf with default values we got results with many strange values
- we first have commented the part [protos], same (wrong) results
- then we have commented [ndpi], then [tuning] ([tuning] seem to be default values), same result
- finally, set to false 'internal', 'external' and 'analysis', here results were correct
what I mean by 'corect' is that the key 'flow_id' to follow detected/update/end-idle is correct to get the hostname and the final size (case: https://github.com/utoni/nDPId/issues/39). but with the default values of the ndpid.conf, the same key 'flow_id' was (many) often the same giving incoherent results as indicated. we suspect the cause is the key 'analysis' that is, if I'm right, a recent option with the libnDPI
right now, we only use that: [general] ... internal=false external=false compression=true analysis=false
other sections are commented from our side.
Fred
more infos:
we have not found all options in the nDPId.c like 'fpc' in the section [ndpi] regarding 'fpc', you use 'disable' but 'libnDPI/doc/configuration_parameters.md' use 'enable'
I've just tested the analyse event within nDPId-test.c. It works on my side.
You may test this as well on your side by changing:
set_cmdarg_boolean(&nDPId_options.enable_data_analysis, 1);
to
set_cmdarg_boolean(&nDPId_options.enable_data_analysis, 0);
in nDPId-test.c:1671.
You may then run ./test/run_tests.sh ./libnDPI ./path/to/nDPId-test and use git diff afterwards to take a closer look what test results have been changed. Only analyse events should disappear.
we have not found all options in the nDPId.c like 'fpc' in the section [ndpi] regarding 'fpc', you use 'disable' but 'libnDPI/doc/configuration_parameters.md' use 'enable'
That's something I'll need to take a closer look. But from my understanding, fpc provides a faster packet classification, but may result in false-positives in some corner cases.
Toni, we have set back your original config, have a look: {"flow_event_id":4,"flow_event_name":"update","thread_id":1,"packet_id":59620,"source":"enp4s0","alias":"ATiBox","flow_id":30,"flow_state":"finished","flow_src_packets_processed":12,"flow_dst_packets_processed":0,"flow_first_seen":1729254260350023,"flow_src_last_pkt_time":1729254319506703,"flow_dst_last_pkt_time":1729254260350023,"flow_idle_time":140000000,"flow_src_min_l4_payload_len":36,"flow_dst_min_l4_payload_len":0,"flow_src_max_l4_payload_len":36,"flow_dst_max_l4_payload_len":0,"flow_src_tot_l4_payload_len":432,"flow_dst_tot_l4_payload_len":0,"midstream":0,"thread_ts_usec":1729254330000712,"l3_proto":"ip4","src_ip":"80.12.193.50","dst_ip":"192.168.4.6","l4_proto":"icmp","flow_datalink":1,"flow_max_packets":15,"ndpi": {"confidence": {"6":"DPI"},"proto":"ICMP","proto_id":"81","proto_by_ip":"Unknown","proto_by_ip_id":0,"encrypted":0,"breed":"Acceptable","category_id":14,"category":"Network"}}
- update: 1729254260350023, 30, ICMP, 432 <- our special line in the our log file, we write the flow_event_name, date, flowid, proto and size
{"flow_event_id":7,"flow_event_name":"detected","thread_id":1,"packet_id":59676,"source":"enp4s0","alias":"ATiBox","flow_id":2096,"flow_state":"info","flow_src_packets_processed":1,"flow_dst_packets_processed":0,"flow_first_seen":1729254330592998,"flow_src_last_pkt_time":1729254330592998,"flow_dst_last_pkt_time":1729254330592998,"flow_idle_time":140000000,"flow_src_min_l4_payload_len":8,"flow_dst_min_l4_payload_len":0,"flow_src_max_l4_payload_len":8,"flow_dst_max_l4_payload_len":0,"flow_src_tot_l4_payload_len":8,"flow_dst_tot_l4_payload_len":0,"midstream":0,"thread_ts_usec":1729254330592998,"l3_proto":"ip4","src_ip":"192.168.4.6","dst_ip":"144.76.59.84","l4_proto":"icmp","ndpi": {"confidence": {"6":"DPI"},"proto":"ICMP","proto_id":"81","proto_by_ip":"Unknown","proto_by_ip_id":0,"encrypted":0,"breed":"Acceptable","category_id":14,"category":"Network"}}
- detected: 1729254330592998, 2096, ICMP, 8
{"flow_event_id":7,"flow_event_name":"detected","thread_id":1,"packet_id":59916,"source":"enp4s0","alias":"ATiBox","flow_id":2150,"flow_state":"info","flow_src_packets_processed":1,"flow_dst_packets_processed":0,"flow_first_seen":1729254332697872,"flow_src_last_pkt_time":1729254332697872,"flow_dst_last_pkt_time":1729254332697872,"flow_idle_time":140000000,"flow_src_min_l4_payload_len":36,"flow_dst_min_l4_payload_len":0,"flow_src_max_l4_payload_len":36,"flow_dst_max_l4_payload_len":0,"flow_src_tot_l4_payload_len":36,"flow_dst_tot_l4_payload_len":0,"midstream":0,"thread_ts_usec":1729254332697872,"l3_proto":"ip4","src_ip":"193.252.159.45","dst_ip":"192.168.4.6","l4_proto":"icmp","ndpi": {"confidence": {"6":"DPI"},"proto":"ICMP","proto_id":"81","proto_by_ip":"Unknown","proto_by_ip_id":0,"encrypted":0,"breed":"Acceptable","category_id":14,"category":"Network","entropy":4.450212}} detected: 1729254332697872, 2150, ICMP, 36
{"flow_event_id":4,"flow_event_name":"update","thread_id":0,"packet_id":60742,"source":"enp4s0","alias":"ATiBox","flow_id":236,"flow_state":"finished","flow_src_packets_processed":2,"flow_dst_packets_processed":0,"flow_first_seen":1729254266339156,"flow_src_last_pkt_time":1729254284862241,"flow_dst_last_pkt_time":1729254266339156,"flow_idle_time":140000000,"flow_src_min_l4_payload_len":69,"flow_dst_min_l4_payload_len":0,"flow_src_max_l4_payload_len":69,"flow_dst_max_l4_payload_len":0,"flow_src_tot_l4_payload_len":138,"flow_dst_tot_l4_payload_len":0,"midstream":0,"thread_ts_usec":1729254339725963,"l3_proto":"ip4","src_ip":"192.168.4.6","dst_ip":"1.1.1.1","l4_proto":"icmp","flow_datalink":1,"flow_max_packets":15,"ndpi": {"confidence": {"6":"DPI"},"proto":"ICMP","proto_id":"81","proto_by_ip":"Unknown","proto_by_ip_id":0,"encrypted":0,"breed":"Acceptable","category_id":14,"category":"Network"}}
- update: 1729254266339156, 236, ICMP, 138
{"flow_event_id":4,"flow_event_name":"update","thread_id":1,"packet_id":60755,"source":"enp4s0","alias":"ATiBox","flow_id":1401,"flow_state":"finished","flow_src_packets_processed":1,"flow_dst_packets_processed":0,"flow_first_seen":1729254306498409,"flow_src_last_pkt_time":1729254306498409,"flow_dst_last_pkt_time":1729254306498409,"flow_idle_time":140000000,"flow_src_min_l4_payload_len":148,"flow_dst_min_l4_payload_len":0,"flow_src_max_l4_payload_len":148,"flow_dst_max_l4_payload_len":0,"flow_src_tot_l4_payload_len":148,"flow_dst_tot_l4_payload_len":0,"midstream":0,"thread_ts_usec":1729254339822064,"l3_proto":"ip4","src_ip":"81.253.129.137","dst_ip":"192.168.4.6","l4_proto":"icmp","flow_datalink":1,"flow_max_packets":15,"ndpi": {"confidence": {"6":"DPI"},"proto":"ICMP","proto_id":"81","proto_by_ip":"Unknown","proto_by_ip_id":0,"encrypted":0,"breed":"Acceptable","category_id":14,"category":"Network"}}
- update: 1729254306498409, 1401, ICMP, 148
it seems the 'flow_id' becomes crazy using very old values when it should and id 2,000+ so after a while, a proto is with a strange (too big) size value.
'fpc' could generate some false-positives, ok but it seems not used by the nDPId.c.
we'll comment option by option in order to find which one is causin the issue, keep you posted...
bye Fred
Toni,
have a look:
ICMP: 1.3MB with 2 hits in the last 7 seconds. something is wrong here :o)
Fred
it seems the 'flow_id' becomes crazy using very old values when it should and id 2,000+ so after a while, a proto is with a strange (too big) size value.
I guess that I understood your problem now. The behavior you're mentioned regarding the flow_id is totally legit. Let's take a close look.
The event name that contains that low/old flow_id is called update. According to the documentation:
update: inform nDPIsrvd or other apps about a long-lasting flow, whose detection was finished a long time ago but is still active
'fpc' could generate some false-positives, ok but it seems not used by the nDPId.c.
That's correct. I am currently testing the fpc behavior.
Toni, have a look:
ICMP: 1.3MB with 2 hits in the last 7 seconds. something is wrong here :o)
Regarding the ICMP flow (also from the README.md)
Other protocols e.g. ICMP/ICMPv6 lack relevance for DPI, thus nDPId does not distinguish between different ICMP/ICMPv6 flows coming from the same host. This saves memory and performance, but might change in the future.
I do not agree, we get ICMP and ICMPv6 from the nDPId, have a look:
update: inform nDPIsrvd or other apps about a long-lasting flow, whose detection was finished a long time ago but is still active
long time ago could mean 2+ hours ago ? hmmm, I'm not sure about that ;o)
Fred
Toni, additional tests done, we suspect the issue is in that part: [protos] tls.certificate_expiration_threshold = 7 tls.application_blocks_tracking = enable stun.max_packets_extra_dissection = 8
original values from libnDPI (libnDPI/doc/configuration_parameters.md) are like that: [protos] tls.certificate_expiration_threshold = 30 tls.application_blocks_tracking = disable stun.max_packets_extra_dissection = 25
using libnDPI values ([protos]), results look correct but we keep a look on data...
Fred
long time ago could mean 2+ hours ago ? hmmm, I'm not sure about that ;o)
An ICMP flow is considered active as long as there is still data on the wire which does belong to that flow. It will time out after icmp-max-idle-time if there was no data seen on the wire during that time. On timeout an idle event will be generated.
original values from libnDPI (libnDPI/doc/configuration_parameters.md) are like that: [protos] tls.certificate_expiration_threshold = 30 tls.application_blocks_tracking = disable stun.max_packets_extra_dissection = 25
I think I'll restore the config file to the default nDPI values.
I'll reopen the issue although it might be better to split the issue into two.
You may now use branch https://github.com/utoni/nDPId/tree/tmp and run ./test/run_config_tests.sh to test certain configurations placed in ./test/configs.