Not logged all data from Interface Device

[HesaMtiPu]

Hi .. we are observing a issue in monster while logging data to click-house from monster. we have two standalone VMs in same location and configured monster with latest version. but in one VMs data are being logged less than second VMs. Supoose we have 100 packets of data then one VMs loogged 80 packets and second VMs logged 30 Packets. I suppose that both VMs should logged full 100 of packets without any loss.

[mosajjal]

Hi. I guess it could be because of your batch size or batch delay settings. change your batch size to 5 and see if that helps with the loss percentge as a test.

[HesaMtiPu]

@mosajjal !!! Loss percentage has come down but full data is not coming yet. Still the capture and output are not equal.

[wthahaha]

2023-11-01T22:21:03+08:00 metrics: {"clickhouseFailed":{"count":0},"clickhouseSentToOutput":{"count":294700000},"clickhouseSkipped":{"count":0},"packetLossPercent":{"value":9.000145739825104},"packetsCaptured":{"value":530609941},"packetsDropped":{"value":47755668},"packetsDuplicate":{"count":0},"packetsOverRatio":{"count":0}}, This is my log,packetLossPercent is 9%, is there any good idea to reduce packetLossPercent ? Thank you. @mosajjal @HesaMtiPu

[mosajjal]

what is your QPS @wthahaha? have you looked at your CPU/RAM usage while running dnsmonster?

[HesaMtiPu]

metrics: {"clickhouseFailed":{"count":0},"clickhouseSentToOutput":{"count":17515},"clickhouseSkipped":{"count":0},"packetLossPercent":{"value":0},"packetsCaptured":{"value":17515},"packetsDropped":{"value":0},"packetsInvalid":{"value":0},"stdoutSentToOutput":{"count":17515},"stdoutSkipped":{"count":0}} @wthahaha PacketLossPercent is 0 and has been tested with data logged with DNStap and a small set of data.