blackbox_exporter
blackbox_exporter copied to clipboard
prometheus
Cannot get TTL from the received packet. 'probe_icmp_reply_hop_limit' will be missing
in fact, use ping to detect, can ping through;
I need to delete the Prometheus node, and then add it back, just fine
error msg：
Logs for the probe: ts=2021-09-14T10:52:30.066887477Z caller=main.go:320 module=icmp target=172.30.0.38 level=info msg="Beginning probe" probe=icmp timeout_seconds=5 ts=2021-09-14T10:52:30.066942749Z caller=icmp.go:91 module=icmp target=172.30.0.38 level=info msg="Resolving target address" ip_protocol=ip4 ts=2021-09-14T10:52:30.066957394Z caller=icmp.go:91 module=icmp target=172.30.0.38 level=info msg="Resolved target address" ip=172.30.0.38 ts=2021-09-14T10:52:30.066965968Z caller=main.go:130 module=icmp target=172.30.0.38 level=info msg="Creating socket" ts=2021-09-14T10:52:30.066982557Z caller=main.go:130 module=icmp target=172.30.0.38 level=debug msg="Unable to do unprivileged listen on socket, will attempt privileged" err="socket: permission denied" ts=2021-09-14T10:52:30.067019185Z caller=main.go:130 module=icmp target=172.30.0.38 level=info msg="Creating ICMP packet" seq=54119 id=19898 ts=2021-09-14T10:52:30.06703098Z caller=main.go:130 module=icmp target=172.30.0.38 level=info msg="Writing out packet" ts=2021-09-14T10:52:30.067096736Z caller=main.go:130 module=icmp target=172.30.0.38 level=info msg="Waiting for reply packets" ts=2021-09-14T10:52:35.067469733Z caller=main.go:130 module=icmp target=172.30.0.38 level=debug msg="Cannot get TTL from the received packet. 'probe_icmp_reply_hop_limit' will be missing." ts=2021-09-14T10:52:35.067502586Z caller=main.go:130 module=icmp target=172.30.0.38 level=warn msg="Timeout reading from socket" err="read ip 0.0.0.0: raw-read ip4 0.0.0.0: i/o timeout" ts=2021-09-14T10:52:35.067536735Z caller=main.go:320 module=icmp target=172.30.0.38 level=error msg="Probe failed" duration_seconds=5.000623111
Metrics that would have been returned:
HELP probe_dns_lookup_time_seconds Returns the time taken for probe dns lookup in seconds
TYPE probe_dns_lookup_time_seconds gauge
probe_dns_lookup_time_seconds 2.3235e-05
HELP probe_duration_seconds Returns how long the probe took to complete in seconds
TYPE probe_duration_seconds gauge
probe_duration_seconds 5.000623111
HELP probe_icmp_duration_seconds Duration of icmp request by phase
TYPE probe_icmp_duration_seconds gauge
probe_icmp_duration_seconds{phase="resolve"} 2.3235e-05 probe_icmp_duration_seconds{phase="rtt"} 0 probe_icmp_duration_seconds{phase="setup"} 6.5027e-05
HELP probe_ip_addr_hash Specifies the hash of IP address. It's useful to detect if the IP address changes.
TYPE probe_ip_addr_hash gauge
probe_ip_addr_hash 4.58557391e+08
HELP probe_ip_protocol Specifies whether probe ip protocol is IP4 or IP6
TYPE probe_ip_protocol gauge
probe_ip_protocol 4
HELP probe_success Displays whether or not the probe was a success
TYPE probe_success gauge
probe_success 0
success msg：
Logs for the probe: ts=2021-09-14T11:17:10.067720519Z caller=main.go:320 module=icmp target=172.30.0.38 level=info msg="Beginning probe" probe=icmp timeout_seconds=5 ts=2021-09-14T11:17:10.067780893Z caller=icmp.go:91 module=icmp target=172.30.0.38 level=info msg="Resolving target address" ip_protocol=ip4 ts=2021-09-14T11:17:10.06779445Z caller=icmp.go:91 module=icmp target=172.30.0.38 level=info msg="Resolved target address" ip=172.30.0.38 ts=2021-09-14T11:17:10.067806344Z caller=main.go:130 module=icmp target=172.30.0.38 level=info msg="Creating socket" ts=2021-09-14T11:17:10.067829115Z caller=main.go:130 module=icmp target=172.30.0.38 level=debug msg="Unable to do unprivileged listen on socket, will attempt privileged" err="socket: permission denied" ts=2021-09-14T11:17:10.067876335Z caller=main.go:130 module=icmp target=172.30.0.38 level=info msg="Creating ICMP packet" seq=7664 id=19898 ts=2021-09-14T11:17:10.06788871Z caller=main.go:130 module=icmp target=172.30.0.38 level=info msg="Writing out packet" ts=2021-09-14T11:17:10.067931151Z caller=main.go:130 module=icmp target=172.30.0.38 level=info msg="Waiting for reply packets" ts=2021-09-14T11:17:10.08316723Z caller=main.go:130 module=icmp target=172.30.0.38 level=info msg="Found matching reply packet" ts=2021-09-14T11:17:10.083205665Z caller=main.go:320 module=icmp target=172.30.0.38 level=info msg="Probe succeeded" duration_seconds=0.015458008
Metrics that would have been returned:
HELP probe_dns_lookup_time_seconds Returns the time taken for probe dns lookup in seconds
TYPE probe_dns_lookup_time_seconds gauge
probe_dns_lookup_time_seconds 2.1311e-05
HELP probe_duration_seconds Returns how long the probe took to complete in seconds
TYPE probe_duration_seconds gauge
probe_duration_seconds 0.015458008
HELP probe_icmp_duration_seconds Duration of icmp request by phase
TYPE probe_icmp_duration_seconds gauge
probe_icmp_duration_seconds{phase="resolve"} 2.1311e-05 probe_icmp_duration_seconds{phase="rtt"} 0.015253655 probe_icmp_duration_seconds{phase="setup"} 8.5602e-05
HELP probe_icmp_reply_hop_limit Replied packet hop limit (TTL for ipv4)
TYPE probe_icmp_reply_hop_limit gauge
probe_icmp_reply_hop_limit 251
HELP probe_ip_addr_hash Specifies the hash of IP address. It's useful to detect if the IP address changes.
TYPE probe_ip_addr_hash gauge
probe_ip_addr_hash 4.58557391e+08
HELP probe_ip_protocol Specifies whether probe ip protocol is IP4 or IP6
TYPE probe_ip_protocol gauge
probe_ip_protocol 4
HELP probe_success Displays whether or not the probe was a success
TYPE probe_success gauge
probe_success 1
Use a lower version will be ok, like blackbox_exporter-0.16.0.linux-amd64
This issue exists for me on a current blackbox_exporter 0.23.0 installation on darwin/arm64:
ts=2023-04-20T02:16:06.890Z caller=handler.go:184 module=icmp target=8.8.8.8 level=debug msg="Creating socket" ts=2023-04-20T02:16:06.892Z caller=handler.go:184 module=icmp target=8.8.8.8 level=debug msg="Creating ICMP packet" seq=64474 id=51526 ts=2023-04-20T02:16:06.892Z caller=handler.go:184 module=icmp target=8.8.8.8 level=debug msg="Writing out packet" ts=2023-04-20T02:16:06.892Z caller=handler.go:184 module=icmp target=8.8.8.8 level=debug msg="Setting TTL (IPv4 unprivileged)" ttl=64 ts=2023-04-20T02:16:06.893Z caller=handler.go:184 module=icmp target=8.8.8.8 level=debug msg="Waiting for reply packets" ts=2023-04-20T02:16:16.378Z caller=handler.go:184 module=icmp target=8.8.8.8 level=debug msg="Cannot get TTL from the received packet. 'probe_icmp_reply_hop_limit' will be missing." ts=2023-04-20T02:16:16.379Z caller=handler.go:184 module=icmp target=8.8.8.8 level=debug msg="Timeout reading from socket" err="read udp 0.0.0.0:0: raw-read udp 0.0.0.0:0: i/o timeout"
The probes are successfully seen by Prometheus. This issue occurs even when blackbox_exporter is run privileged.
M1 Max running Ventura 13.3.1
This is still present in 0.24.0.
unprivileged:
ts=2023-06-07T13:18:17.056187Z caller=main.go:181 module=icmp target=8.8.8.8 level=info msg="Beginning probe" probe=icmp timeout_seconds=9.5
ts=2023-06-07T13:18:17.056316Z caller=icmp.go:91 module=icmp target=8.8.8.8 level=info msg="Resolving target address" target=8.8.8.8 ip_protocol=ip4
ts=2023-06-07T13:18:17.056334Z caller=icmp.go:91 module=icmp target=8.8.8.8 level=info msg="Resolved target address" target=8.8.8.8 ip=8.8.8.8
ts=2023-06-07T13:18:17.05635Z caller=handler.go:120 module=icmp target=8.8.8.8 level=info msg="Creating socket"
ts=2023-06-07T13:18:17.056461Z caller=handler.go:120 module=icmp target=8.8.8.8 level=info msg="Creating ICMP packet" seq=22284 id=22023
ts=2023-06-07T13:18:17.056485Z caller=handler.go:120 module=icmp target=8.8.8.8 level=info msg="Writing out packet"
ts=2023-06-07T13:18:17.056499Z caller=handler.go:120 module=icmp target=8.8.8.8 level=debug msg="Setting TTL (IPv4 unprivileged)" ttl=64
ts=2023-06-07T13:18:17.056626Z caller=handler.go:120 module=icmp target=8.8.8.8 level=info msg="Waiting for reply packets"
ts=2023-06-07T13:18:26.558976Z caller=handler.go:120 module=icmp target=8.8.8.8 level=debug msg="Cannot get TTL from the received packet. 'probe_icmp_reply_hop_limit' will be missing."
ts=2023-06-07T13:18:26.559107Z caller=handler.go:120 module=icmp target=8.8.8.8 level=warn msg="Timeout reading from socket" err="read udp 0.0.0.0:0: raw-read udp 0.0.0.0:0: i/o timeout"
ts=2023-06-07T13:18:26.55959Z caller=main.go:181 module=icmp target=8.8.8.8 level=error msg="Probe failed" duration_seconds=9.503360958
Privileged
ts=2023-06-07T13:19:32.056638Z caller=main.go:181 module=icmp target=8.8.8.8 level=info msg="Beginning probe" probe=icmp timeout_seconds=9.5
ts=2023-06-07T13:19:32.056789Z caller=icmp.go:91 module=icmp target=8.8.8.8 level=info msg="Resolving target address" target=8.8.8.8 ip_protocol=ip4
ts=2023-06-07T13:19:32.056817Z caller=icmp.go:91 module=icmp target=8.8.8.8 level=info msg="Resolved target address" target=8.8.8.8 ip=8.8.8.8
ts=2023-06-07T13:19:32.056839Z caller=handler.go:120 module=icmp target=8.8.8.8 level=info msg="Creating socket"
ts=2023-06-07T13:19:32.057017Z caller=handler.go:120 module=icmp target=8.8.8.8 level=info msg="Creating ICMP packet" seq=14730 id=22104
ts=2023-06-07T13:19:32.057054Z caller=handler.go:120 module=icmp target=8.8.8.8 level=info msg="Writing out packet"
ts=2023-06-07T13:19:32.057079Z caller=handler.go:120 module=icmp target=8.8.8.8 level=debug msg="Setting TTL (IPv4 unprivileged)" ttl=64
ts=2023-06-07T13:19:32.057833Z caller=handler.go:120 module=icmp target=8.8.8.8 level=info msg="Waiting for reply packets"
ts=2023-06-07T13:19:41.5578Z caller=handler.go:120 module=icmp target=8.8.8.8 level=debug msg="Cannot get TTL from the received packet. 'probe_icmp_reply_hop_limit' will be missing."
ts=2023-06-07T13:19:41.557922Z caller=handler.go:120 module=icmp target=8.8.8.8 level=warn msg="Timeout reading from socket" err="read udp 0.0.0.0:0: raw-read udp 0.0.0.0:0: i/o timeout"
ts=2023-06-07T13:19:41.558166Z caller=main.go:181 module=icmp target=8.8.8.8 level=error msg="Probe failed" duration_seconds=9.501469291
Relevant configuration:
icmp:
prober: icmp
icmp:
preferred_ip_protocol: ip4
ip_protocol_fallback: true
icmp_ttl5:
prober: icmp
timeout: 5s
icmp:
ttl: 5
Right, so. If you set dont_fragment (which forces the ICMP probe to run privileged), the RTT is returned correctly. This is very likely related directly to https://github.com/golang/go/issues/47369 and should be fixed when upstream is fixed.
I'm experiencing this issue and the dont_fragment option has no positive effect on those IPs affected by this.
What's interesting is this. The probes intermittently fail from Prometheus triggering the test for certain IPs, but if I run the test manually in a web browser like below, it always returns probe_success == 1.
https://blackbox_exporter/blackbox/probe?target=10.140.33.23&module=icmp&debug=true
icmp: prober: icmp timeout: 5s icmp: dont_fragment: true preferred_ip_protocol: "ip4"
I'm experiencing this issue and the dont_fragment option has no positive effect on those IPs affected by this.
What's interesting is this. The probes intermittently fail from Prometheus triggering the test for certain IPs, but if I run the test manually in a web browser like below, it always returns probe_success == 1.
https://blackbox_exporter/blackbox/probe?target=10.140.33.23&module=icmp&debug=true
icmp: prober: icmp timeout: 5s icmp: dont_fragment: true preferred_ip_protocol: "ip4"
The probe is always successful, but the RTT isn't reported (or at least isn't on my box). Setting dont_fragment only has an effect if your probe is run privileged. You have to do both.
silly question maybe, but I'm running the exporter container v0.22.0
the binary is run by root in the container so it's running privileged, no? or is the Prometheus job the issue if that container is not running in privileged mode?
Logs for the probe:
ts=2023-08-09T14:19:54.943527595Z caller=main.go:175 module=icmp target=10.140.33.23 level=info msg="Beginning probe" probe=icmp timeout_seconds=4.5
ts=2023-08-09T14:19:54.943572693Z caller=icmp.go:91 module=icmp target=10.140.33.23 level=info msg="Resolving target address" target=10.140.33.23 ip_protocol=ip4
ts=2023-08-09T14:19:54.943581835Z caller=icmp.go:91 module=icmp target=10.140.33.23 level=info msg="Resolved target address" target=10.140.33.23 ip=10.140.33.23
ts=2023-08-09T14:19:54.943589331Z caller=handler.go:117 module=icmp target=10.140.33.23 level=info msg="Creating socket"
ts=2023-08-09T14:19:54.943639758Z caller=handler.go:117 module=icmp target=10.140.33.23 level=info msg="Creating ICMP packet" seq=54339 id=57225
ts=2023-08-09T14:19:54.943651774Z caller=handler.go:117 module=icmp target=10.140.33.23 level=info msg="Writing out packet"
ts=2023-08-09T14:19:54.943659615Z caller=handler.go:117 module=icmp target=10.140.33.23 level=debug msg="Overriding TTL (raw IPv4)" ttl=64
ts=2023-08-09T14:19:54.943735443Z caller=handler.go:117 module=icmp target=10.140.33.23 level=info msg="Waiting for reply packets"
ts=2023-08-09T14:19:59.444459664Z caller=handler.go:117 module=icmp target=10.140.33.23 level=debug msg="Cannot get TTL from the received packet. 'probe_icmp_reply_hop_limit' will be missing."
ts=2023-08-09T14:19:59.444509566Z caller=handler.go:117 module=icmp target=10.140.33.23 level=warn msg="Timeout reading from socket" err="read ip 0.0.0.0: raw-read ip4 0.0.0.0: i/o timeout"
ts=2023-08-09T14:19:59.444577666Z caller=main.go:175 module=icmp target=10.140.33.23 level=error msg="Probe failed" duration_seconds=4.501033673
Same issue here. Can sb help?
caller=main.go:181 module=icmp target=192.168.10.10 level=info msg="Beginning probe" probe=icmp timeout_seconds=5
caller=icmp.go:91 module=icmp target=192.168.10.10 level=info msg="Resolving target address" target=192.168.10.10 ip_protocol=ip4
caller=icmp.go:91 module=icmp target=192.168.10.10 level=info msg="Resolved target address" target=192.168.10.10 ip=192.168.10.10
caller=handler.go:120 module=icmp target=192.168.10.10 level=info msg="Creating socket"
caller=handler.go:120 module=icmp target=192.168.10.10 level=info msg="Creating ICMP packet" seq=46525 id=60779
caller=handler.go:120 module=icmp target=192.168.10.10 level=info msg="Writing out packet"
caller=handler.go:120 module=icmp target=192.168.10.10 level=debug msg="Overriding TTL (raw IPv4)" ttl=64
caller=handler.go:120 module=icmp target=192.168.10.10 level=info msg="Waiting for reply packets"
caller=handler.go:120 module=icmp target=192.168.10.10 level=debug msg="Cannot get TTL from the received packet. 'probe_icmp_reply_hop_limit' will be missing."
caller=handler.go:120 module=icmp target=192.168.10.10 level=warn msg="Timeout reading from socket" err="read ip 0.0.0.0: raw-read ip4 0.0.0.0: i/o timeout"
caller=main.go:181 module=icmp target=192.168.10.10 level=error msg="Probe failed" duration_seconds=5.000325715
blackbox exporter config:
Module configuration:
prober: icmp
timeout: 5s
http:
ip_protocol_fallback: true
follow_redirects: true
enable_http2: true
tcp:
ip_protocol_fallback: true
icmp:
preferred_ip_protocol: ip4
ip_protocol_fallback: true
dont_fragment: true
ttl: 64
dns:
values.yml:
securityContext:
runAsUser: 0
runAsGroup: 0
readOnlyRootFilesystem: true
runAsNonRoot: false
allowPrivilegeEscalation: true
capabilities:
add: ["NET_RAW"]
