blackbox_exporter Log level for unsuccessful probes

We have update to v.0.26 and now observing the situation with probe logs. And these observations show some logic we are not sure about. If we are probing some target that the exporter can not resolve - we get an ERROR level log, e.g.

Mar 30 10:44:31 prometheus-test docker[658698]: time=2025-03-30T08:44:31.841Z level=ERROR source=utils.go:86 msg="Resolution with IP protocol failed" module=icmp target=fake_target.example.net target=fake_target.example.net err="lookup fake_target.example.net on 172.0.0.44:53: no such host"

Ok. And now we probing some existing target, but get not expected status code and probe is failing. But exporter logging the reason for the failed probe with INFO level, e.g.

2025-03-30 11:08:01.634 
time=2025-03-30T08:08:01.633Z level=INFO source=http.go:488 msg="Invalid HTTP response status code" module=http_2xx_no_verify target=https://existing.target.com/ status_code=403 valid_status_codes=[200]

So we can not start the exporter with --probe.level=warn to get only failed probes, and we have to start it with --probe.level=info which produces millions of logs for successful probes. So the question is if the reason of probe failing could be logged with ERROR/WARN level?

Apr 04 '25 03:04 winadm87

cc @tjhop

Apr 04 '25 03:04 winadm87

We would be interested in this as well, getting HTTP status codes for failures on warn/error probe.level.

Apr 09 '25 09:04 Abhinand-Gokul

There's a related conversation to the log prober logging in #1377, I've asked for some clarification there and would like to make a determination on how to proceed there before attempting to tune leveling further

Apr 10 '25 03:04 tjhop

@tjhop sorry for mentioning, but we continue to struggle with logic in probe logs. We updated to v.0.27 and now we can control on which level all the probe logs are written. So we are doing this

      - '--log.level=info'
      - '--log.prober=warn'

we are getting this in the logs

blackbox-exporter-1  | time=2025-08-05T03:06:52.300Z level=INFO source=main.go:88 msg="Starting blackbox_exporter" version="(version=0.27.0, branch=HEAD, revision=46ea0224f57b708fdff317f8872765a77ef64034)"
blackbox-exporter-1  | time=2025-08-05T03:06:52.300Z level=INFO source=main.go:89 msg="(go=go1.24.4, platform=linux/arm64, user=root@2d9f1d02ac3f, date=20250630-12:37:26, tags=unknown)"
blackbox-exporter-1  | time=2025-08-05T03:06:52.301Z level=INFO source=main.go:101 msg="Loaded config file"
blackbox-exporter-1  | time=2025-08-05T03:06:52.302Z level=INFO source=tls_config.go:347 msg="Listening on" address=[::]:9115
blackbox-exporter-1  | time=2025-08-05T03:06:52.302Z level=INFO source=tls_config.go:350 msg="TLS is disabled." http2=false address=[::]:9115
blackbox-exporter-1  | time=2025-08-05T03:07:15.069Z level=WARN source=handler.go:194 msg="Beginning probe" module=icmp target=google.com probe=icmp timeout_seconds=119.5
blackbox-exporter-1  | time=2025-08-05T03:07:15.069Z level=WARN source=handler.go:194 msg="Resolving target address" module=icmp target=google.com target=google.com ip_protocol=ip4
blackbox-exporter-1  | time=2025-08-05T03:07:15.164Z level=WARN source=handler.go:194 msg="Resolved target address" module=icmp target=google.com target=google.com ip=142.250.179.206
blackbox-exporter-1  | time=2025-08-05T03:07:15.164Z level=WARN source=handler.go:194 msg="Creating socket" module=icmp target=google.com
blackbox-exporter-1  | time=2025-08-05T03:07:15.164Z level=WARN source=handler.go:194 msg="Creating ICMP packet" module=icmp target=google.com seq=64212 id=25791
blackbox-exporter-1  | time=2025-08-05T03:07:15.164Z level=WARN source=handler.go:194 msg="Writing out packet" module=icmp target=google.com
blackbox-exporter-1  | time=2025-08-05T03:07:15.165Z level=WARN source=handler.go:194 msg="Waiting for reply packets" module=icmp target=google.com
blackbox-exporter-1  | time=2025-08-05T03:07:15.245Z level=WARN source=handler.go:194 msg="Found matching reply packet" module=icmp target=google.com
blackbox-exporter-1  | time=2025-08-05T03:07:15.245Z level=WARN source=handler.go:194 msg="Probe succeeded" module=icmp target=google.com duration_seconds=0.17611625

How can we utilize such a log switch configuration if we want to write to logs for probes only errors, warnings and failes? Like if probe succeed - dont write to log anything about this probe, but if probe failed - write all the strings about this probe with warn/error level? Is this possible somehow?

Aug 05 '25 03:08 winadm87

we made some changes in the way logger works in PR: https://github.com/prometheus/blackbox_exporter/pull/1461, and even updated some log levels for some logs lines.

now --log.prober and --log.level are not coupled and should work as expected. Please try running build from master, and report back. imo, this should be fixed on latest master.

Oct 29 '25 19:10 electron0zero

yep, builded from master, checked the result and with default logging parameters got expected result!

docker logs zealous_kare -f
time=2025-10-30T11:21:28.858Z level=INFO source=main.go:98 msg="Starting blackbox_exporter" version="(version=0.27.0, branch=master, revision=9922360ebc1a7fe7dc8cd2ea6efacd6176639e9d)"
time=2025-10-30T11:21:28.863Z level=INFO source=main.go:99 msg="(go=go1.25.3, platform=linux/amd64, user=admin@machine date=20251030-11:14:40, tags=unknown)"
time=2025-10-30T11:21:28.863Z level=INFO source=config.go:135 msg="Configuration file change detected, reloading the configuration."
time=2025-10-30T11:21:28.872Z level=WARN source=config.go:162 msg="HTTP/3 is enabled for this module. HTTP targets will be automatically converted to HTTPS during probing. Consider using HTTPS targets directly in your configuration." module=http_3xx
time=2025-10-30T11:21:28.873Z level=INFO source=main.go:111 msg="Loaded config file"
time=2025-10-30T11:21:28.885Z level=INFO source=tls_config.go:346 msg="Listening on" address=[::]:9115
time=2025-10-30T11:21:28.885Z level=INFO source=tls_config.go:349 msg="TLS is disabled." http2=false address=[::]:9115
time=2025-10-30T11:21:57.325Z level=WARN source=http.go:490 msg="Received redirect" module=http_2xx target=prometheus.io location=https://prometheus.io/
time=2025-10-30T11:22:16.547Z level=WARN source=http.go:490 msg="Received redirect" module=http_2xx target=google.com location=http://www.google.com/
time=2025-10-30T11:22:37.479Z level=ERROR source=utils.go:86 msg="Resolution with IP protocol failed" module=http_2xx target=notexistingtarget.com target=notexistingtarget.com err="lookup notexistingtarget.com on 192.168.65.7:53: no such host"
time=2025-10-30T11:22:37.480Z level=ERROR source=http.go:410 msg="Error resolving address" module=http_2xx target=notexistingtarget.com err="lookup notexistingtarget.com on 192.168.65.7:53: no such host"
time=2025-10-30T11:22:37.481Z level=ERROR source=handler.go:130 msg="Probe failed" module=http_2xx target=notexistingtarget.com duration_seconds=0.194954416

@tjhop thank you very much for your work!

Oct 30 '25 11:10 winadm87