nProbe icon indicating copy to clipboard operation
nProbe copied to clipboard

nprobe not listening on zmq port after reboot

Open fschlich opened this issue 7 months ago • 0 comments

I run nprobe (from the Debian package) in collector mode exporting flows for ntopng, using the following options:

-i=bond0
--ntopng=zmq://10.254.100.252:5556
-n=none
-T=@NTOPNG@

After a reboot, I regularly find that nprobe is running, but hasn't opened port 5556 and no flows are visible in ntopng:

Mar 11 16:55:14 myhost nprobe[2185]: 11/Mar/2025 16:55:14 [nprobe.c:8346] Welcome to nProbe v.10.6.240906 for x86_64-pc-linux-gnu with native PF_RING acceleration
Mar 11 16:55:14 myhost nprobe[2185]: 11/Mar/2025 16:55:14 [nprobe.c:8368] Pro Edition running on Debian GNU/Linux 11 (bullseye)
...
Mar 11 16:55:14 myhost nprobe[2185]: 11/Mar/2025 16:55:14 [util.c:6659] Initializing ZMQ as server
Mar 11 16:55:14 myhost nprobe[2185]: 11/Mar/2025 16:55:14 [util.c:6694] ERROR: Unable to bind ZMQ endpoint tcp://10.254.100.252:5556: Cannot assign requested address
Mar 11 16:55:14 myhost nprobe[2185]: 11/Mar/2025 16:55:14 [pf_ring.c:434] Initializing PF_RING socket on device bond0.. (promisc)
Mar 11 16:55:14 myhost nprobe[2185]: 11/Mar/2025 16:55:14 [nprobe.c:9482] Initializing pcap socket on device bond0 (promisc)
Mar 11 16:55:14 myhost nprobe[2185]: 11/Mar/2025 16:55:14 [nprobe.c:12565] Capturing packets from interface bond0 [snaplen: 1600 bytes]
Mar 11 16:55:14 myhost nprobe[2185]: 11/Mar/2025 16:55:14 [util.c:5463] nProbe changed user to 'nprobe'
Mar 11 16:55:14 myhost nprobe[2185]: 11/Mar/2025 16:55:14 [export.c:487] Using TLV as serialization format
Mar 11 16:55:14 myhost nprobe[2185]: 11/Mar/2025 16:55:14 [nprobe.c:12876] nProbe started successfully

I see two problems here:

  1. if the ZMQ server cannot be started, nprobe should abort with an error instead of going on and claiming to have started successfully

  2. the reason it can't bind to the requested IP address is that the interface isn't fully configured yet. The systemd service shipped by the official Debian package depends on network.target, when in fact it should depend on network-online.target in order to be able to bind to a particular IP address. See https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/

Restarting [email protected] after I'm able to login via SSH fixes the problem, as does adding an override for the systemd unit that adds After/Wants=network-online.target, but this is a bandaid and should be fixed upstream in the package. If nprobe were to properly fail in such circumstances, we would at least get an alert from our monitoring system instead of stumbling over an empty ntopng days later.

fschlich avatar Mar 13 '25 21:03 fschlich