network-traffic-metrics
network-traffic-metrics copied to clipboard
zaneclaes
IP is address wrongly captured
When I run py script as sudo python3 /home/pi/network-traffic-metrics/network-traffic-metrics.py "(src net 192.168.8.0/24 and not dst net 192.168.8.0/24) or (dst net 192.168.8.0/24 and not src net 192.168.8.0/24)"
I see here http://192.168.8.1:8000/metrics the next wrongly parsed IP. Mostly for ICMP proto
ntm_packets_total{dst="168.8",proto="icmp",service="",src="cust.tele2"} 8.0 ntm_packets_total{dst="168.8",proto="icmp",service="",src="59.49"} 1.0 ntm_packets_total{dst="168.8",proto="icmp",service="",src="6.3sbsdatacom-pvt-ltd"} 1.0 ntm_packets_total{dst="168.8",proto="icmp",service="",src="static-31-148-113-93.netbynet"} 2.0 ntm_packets_total{dst="168.8",proto="icmp",service="",src="108.216"} 1.0 ntm_packets_total{dst="igmp.mcast",proto="igmp",service="",src="168.8"} 11319.0
Also, because of such script behavior, not all traffic is reflected in Grafana
Possibilities:
- Your IP addresses do not match the regex
[\w\d\.-]+, and therefore are not valid IPs - The full IP is not reported by
tcpdump, indicating a config problem with your network or filters. - Some other strange feature of your network is breaking the regx matching.
I'd try running tcpdump directly (using the same parameters) and examining the output to see why it's not matching. If you want to post the output here I can take a look.
I found specific (2-lines) tcpdump example with icmp protocol that could be parsed (as I assume) as "168.8" Those lines looks like: 20:04:07.887449 IP (tos 0xc0, ttl 64, id 6849, offset 0, flags [none], proto ICMP (1), length 157) 46.175.232.60.maxnet.net.pl > 192.168.8.101: ICMP host 46.175.232.60.maxnet.net.pl unreachable - admin prohibited, length 137 IP (tos 0x0, ttl 64, id 33875, offset 0, flags [DF], proto UDP (17), length 129) 192.168.8.101.6881 > 46.175.232.60.maxnet.net.pl.6881: UDP, length 101[|icmp]
I'm not sure that I need sush info in the report, because it contains "unreachable" word
The "normal" (1-line) output from tcpdump usually looks like: 20:04:08.012908 IP (tos 0x0, ttl 64, id 63617, offset 0, flags [DF], proto UDP (17), length 129) 192.168.8.101.6881 > 106.207.232.60.59969: UDP, length 101
Yeah, that sounds right. In particular, I think it's the - admin prohibited message that breaks it. I've never seen a message like that before. I'm sure we could fix the regex, but I'm wondering if this behavior is expected. Is there some sort of firewall or something causing this?
On my Rspi 4 only standard firewall was enabled. As per your instructions
Hm. everything I can find says that message is caused by a firewall. Keep in mind, firewalls don't only live on your Raspberry Pi. If there's anything upstream of it, the firewall might live there. Do you recognize the maxnet.net.pl URL? It looks like it may be a Polish ISP, in which case I wonder if this is on their end.
Okay, here's what I'm seeing then. The line which is breaking things is:
46.175.232.60.maxnet.net.pl > 192.168.8.101: ICMP host 46.175.232.60.maxnet.net.pl unreachable - admin prohibited, length 137
IP (tos 0x0, ttl 64, id 33875, offset 0, flags [DF], proto UDP (17), length 129)
(Okay, two lines). This indicates that the ISP (maxnet) is trying to communicate with your device at 192.168.8.101, but is being prevented from doing so. Which sounds very much like an ingress firewall to me, and may indicate packet loss in the communication between you and your ISP.
In the link from the prior post, he suggested running systemctl stop firewalld.service. This is a good way to test for a short period of time and see if the error goes away. If it does, great, now find the config problem. If not, I'd try making 100% certain you don't have a different firewall installed, as well. For example, the official Raspberry Pi docs recommend UFW, which is a different firewall and would conflict with firewalld (which I had recommended). Note that there is no real "standard" firewall.
Btw, if you need a work-around for now, you could either (1) adjust the script parameters to filter the offending IPs/URLs or (2) filter them after the fact in Grafana. However, the fact that you're seeing this block is concerning w.r.t. your actual internet performance (re: packet loss, etc.)
I'm seeing the same behaviour w/ ICMP. I think the main issue is that ICMP packets don't have a protocol string or port number after the IP address.
e.g. The lines being parsed by the regex are similar to:
(tos 0x0, ttl 64, id 64081, offset 0, flags [DF], proto ICMP (1), length 84)
10.11.69.35 > dns.google: ICMP echo request, id 7695, seq 6174, length 64
The above is parsed to the following label set: {'src': '11.69', 'dst': 'dns', 'proto': 'icmp', 'service': None}
The regex seems to parse the address into 10.11.69, treating the last octet (35) as the source port. After going through the extract_domain() function, it further gets shortened to just 11.69. I'm not a regex expert, but this seems a bit difficult to fix without further complicating the regex... perhaps adding an exception to ICMP may be an easier solution? But it's not very clean/elegant.
@zaneclaes Pls prevent tcpdump from resolving ip addresses by adding the -n argungent.
https://wiki.ubuntuusers.de/tcpdump/
