blackbox_exporter
blackbox_exporter copied to clipboard
prometheus
Blackbox exporter returns status code 404 for available endpoints , which are returning status code 200
Host operating system: output of uname -a
Its used from the official helm chart and docker image.
blackbox_exporter version: output of blackbox_exporter --version
blackbox_exporter, version 0.19.0 (branch: HEAD, revision: 5d575b88eb12c65720862e8ad2c5890ba33d1ed0) build user: root@2b0258d5a55a build date: 20210510-12:56:44 go version: go1.16.4 platform: linux/amd64
What is the blackbox.yml module config.
prometheus-blackbox-exporter:
config:
modules:
# module used for standard 2xx http ping
http_2xx:
http:
no_follow_redirects: false
preferred_ip_protocol: ip4
tls_config:
insecure_skip_verify: true
valid_http_versions:
- HTTP/1.1
- HTTP/2
prober: http
timeout: 5s
What is the prometheus.yml scrape config.
- job_name: 'blackbox'
metrics_path: /probe
params:
module: [http_2xx]
static_configs:
- targets:
- https://my.url.com
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: prometheus-blackbox-exporter.monitoring.svc.local:9115
What logging output did you get from adding &debug=true to the probe URL?
ts=2022-09-07T13:05:32.068871581Z caller=main.go:320 module=http_2xx target=https://my.url.com level=info msg="Beginning probe" probe=http timeout_seconds=5
ts=2022-09-07T13:05:32.069504382Z caller=http.go:335 module=http_2xx target=https://my.url.com level=info msg="Resolving target address" ip_protocol=ip4
ts=2022-09-07T13:05:32.079432156Z caller=http.go:335 module=http_2xx target=https://my.url.com level=info msg="Resolved target address" ip=167.312.321.123 -> (my lb ip)
ts=2022-09-07T13:05:32.079534865Z caller=client.go:251 module=http_2xx target=https://my.url.com level=info msg="Making HTTP request" url=167.312.321.123 -> (my lb ip) host=my.url.com
ts=2022-09-07T13:05:32.087609003Z caller=main.go:130 module=http_2xx target=https://my.url.com level=info msg="Received HTTP response" status_code=404
ts=2022-09-07T13:05:32.087658114Z caller=main.go:130 module=http_2xx target=https://my.url.com level=info msg="Invalid HTTP response status code, wanted 2xx" status_code=404
ts=2022-09-07T13:05:32.087953971Z caller=main.go:130 module=http_2xx target=https://my.url.com level=error msg="Invalid HTTP version number" version=HTTP/2.0
ts=2022-09-07T13:05:32.088058603Z caller=main.go:130 module=http_2xx target=https://my.url.com level=info msg="Response timings for roundtrip" roundtrip=0 start=2022-09-07T13:05:32.079624092Z dnsDone=2022-09-07T13:05:32.079624092Z connectDone=2022-09-07T13:05:32.08060517Z gotConn=2022-09-07T13:05:32.085857596Z responseStart=2022-09-07T13:05:32.08755494Z tlsStart=2022-09-07T13:05:32.080633552Z tlsDone=2022-09-07T13:05:32.085654131Z end=2022-09-07T13:05:32.087935858Z
ts=2022-09-07T13:05:32.088584557Z caller=main.go:320 module=http_2xx target=https://my.url.com level=error msg="Probe failed" duration_seconds=0.019597321
Metrics that would have been returned:
probe_dns_lookup_time_seconds 0.009964898
probe_duration_seconds 0.019597321
probe_failed_due_to_regex 0
probe_http_content_length 19
probe_http_duration_seconds{phase="connect"} 0.000981106
probe_http_duration_seconds{phase="processing"} 0.001697285
probe_http_duration_seconds{phase="resolve"} 0.009964898
probe_http_duration_seconds{phase="tls"} 0.005020672
probe_http_duration_seconds{phase="transfer"} 0.000380997
probe_http_redirects 0
probe_http_ssl 1
probe_http_status_code 404
probe_http_uncompressed_body_length 19
probe_http_version 2
probe_ip_addr_hash 7.88769462e+08
probe_ip_protocol 4
probe_ssl_earliest_cert_expiry 1.694091712e+09
probe_ssl_last_chain_expiry_timestamp_seconds -6.21355968e+10
probe_ssl_last_chain_info{fingerprint_sha256="ae66cb45d40024051b6d42a3f46ea6ab67148ce37115bb223aba219f2e1da873"} 1
probe_success 0
probe_tls_version_info{version="TLS 1.3"} 1
Metrics that would have been returned:
probe_dns_lookup_time_seconds 0.009964898
probe_duration_seconds 0.019597321
probe_failed_due_to_regex 0
probe_http_content_length 19
probe_http_duration_seconds{phase="connect"} 0.000981106
probe_http_duration_seconds{phase="processing"} 0.001697285
probe_http_duration_seconds{phase="resolve"} 0.009964898
probe_http_duration_seconds{phase="tls"} 0.005020672
probe_http_duration_seconds{phase="transfer"} 0.000380997
probe_http_redirects 0
probe_http_ssl 1
probe_http_status_code 404
probe_http_uncompressed_body_length 19
probe_http_version 2
probe_ip_addr_hash 7.88769462e+08
probe_ip_protocol 4
probe_ssl_earliest_cert_expiry 1.694091712e+09
probe_ssl_last_chain_expiry_timestamp_seconds -6.21355968e+10
probe_ssl_last_chain_info{fingerprint_sha256="ae66cb45d40024051b6d42a3f46ea6ab67148ce37115bb223aba219f2e1da873"} 1
probe_success 0
probe_tls_version_info{version="TLS 1.3"} 1
What did you do that produced an error?
Probe returns 404 , as curl and browsers are working without issues.
What did you expect to see?
Status code 200
What did you see instead?
Status code 404
I tried all possible configs as promtheus kubernetes_sd_configs for the ingress , MULTI-TARGET EXPORTER PATTERN , static targets. Every time it gives 404 , as we are using traefik and hetzner lb. Working curl example : curl -X GET -k -H "Host: https://my.url.com" -vvv https://167.312.321.123 -> (my lb ip)
+1 Here, I was wondering if there is a previous release that did not have this problem? So that we can investigate on what code change caused this and maybe fix it/roll it back?
blackbox_exporter 0.19.0 is several versions old. Is this still reproducible with v0.23.0?
It appears that the correct path is not /probe but rather /[hostname]/probe. Why is this?
This appears to possibly be a result of --web.external-url= in my case. Setting --web.route-prefix=/ solved the issue for me.
blackbox_exporter 0.19.0 is several versions old. Is this still reproducible with v0.23.0?
having similar issue in v0.24.0
I'm not able to reproduce this by probing e.g. https://google.com/.
Are you only getting unexpected 404 responses from specific targets, or all targets? If only specific targets, then it would be helpful to know what they are, as it may suggest that the issue lies more with the target than with blackbox_exporter.
I'm not able to reproduce this by probing e.g. https://google.com/.
Are you only getting unexpected 404 responses from specific targets, or all targets? If only specific targets, then it would be helpful to know what they are, as it may suggest that the issue lies more with the target than with blackbox_exporter.
In my case, it is only specific targets. I believe my issue is with nginx or SSL. i have cleaned up my nginx host config file and renewed my SSL using certbot then after restarting nginx, it works