http: Both A and AAAA records are queried in redirects regardless of preferred_ip_protocol/ip_protocol_fallback

[jgosmann]

When probing an endpoint that redirects to a different URL, the both the A and AAAA record are queried for the URL being redirected to. The preferred_ip_protocol and ip_protocol_fallback are not considered.

Previously this issue has been reported also for the initial HTTP request (#724) and been fixed in (#728).

Let us consider the following example:

• http://example.com:8080/example redirects to /example/ (resulting URL http://example:8080/example/)
• http://example.com:8080/example/ redirects to https://example.com:8443/example/

Then the blackbox_exporter will first resolve example.com and only query the A record because preferred_ip_protocol and ip_protocol_fallback are considered correctly (source). Then the hostname is replaced with the IP in the URL (source). The subsequent HTTP request will return a redirect with path only (to /example/). Thus, the IP is kept as the hostname and no additional DNS query is made yet. The next HTTP query (to http://example.com:8080/example/) redirects to a full URL (https://example.com:8443/example/). This redirect (as the first one) is handled by the HTTP client that does not know about the desired protocol and will query both A and AAAA whereas only the A query is desired.

This is a problem as some firewalls seem to block the AAAA response if it is too late after the A response. Then the DNS resolution will wait for a timeout (~5s) and probing the URL might run into a timeout too even though it would be reachable.

Host operating system: output of uname -a

Linux [redacted] 4.18.0-348.20.1.el8_5.x86_64 #1 SMP Thu Mar 10 20:59:28 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

blackbox_exporter version: output of blackbox_exporter --version

blackbox_exporter, version 0.21.1 (branch: HEAD, revision: e0d5e4453c54b6e508531a761131d7f21007783c)
  build user:       root@6e3666c7d63f
  build date:       20220617-12:22:49
  go version:       go1.18.3
  platform:         linux/amd64

What is the blackbox.yml module config.

modules:
  https_ipv4_status_200:
    prober: http
    timeout: 5s
    http:
      valid_status_codes:
      - 200
      follow_redirects: true
      fail_if_not_ssl: true
      preferred_ip_protocol: ip4
      ip_protocol_fallback: false
      tls_config:
        insecure_skip_verify: true
        min_version: TLS12

What is the prometheus.yml scrape config.

irrelevant, can be reproduced by manually scraping with:

curl 'localhost:9115/probe?module=https_ipv4_status_200&target=http://example.com:8080/example'

What logging output did you get from adding &debug=true to the probe URL?

ts=2023-01-24T11:42:50.732492717Z caller=main.go:351 module=https_ipv4_status_200 target=http://example.com:8080/example level=info msg="Beginning probe" probe=http timeout_seconds=5
ts=2023-01-24T11:42:50.733397798Z caller=http.go:336 module=https_ipv4_status_200 target=http://example.com:8080/example level=info msg="Resolving target address" target=example.com ip_protocol=ip4
ts=2023-01-24T11:42:50.744752374Z caller=http.go:336 module=https_ipv4_status_200 target=http://example.com:8080/example level=info msg="Resolved target address" target=example.com ip=192.0.2.1
ts=2023-01-24T11:42:50.745066911Z caller=client.go:252 module=https_ipv4_status_200 target=http://example.com:8080/example level=info msg="Making HTTP request" url=http://192.0.2.1:8080/example host=example.com:8080
ts=2023-01-24T11:42:50.747330548Z caller=client.go:498 module=https_ipv4_status_200 target=http://example.com:8080/example level=info msg="Received redirect" location=/example/
ts=2023-01-24T11:42:50.747607444Z caller=client.go:252 module=https_ipv4_status_200 target=http://example.com:8080/example level=info msg="Making HTTP request" url=http://192.0.2.1:8080/example/ host=example.com:8080
ts=2023-01-24T11:42:50.74933535Z caller=client.go:498 module=https_ipv4_status_200 target=http://example.com:8080/example level=info msg="Received redirect" location=https://example.com:8443/example/
ts=2023-01-24T11:42:50.749559852Z caller=client.go:252 module=https_ipv4_status_200 target=http://example.com:8080/example level=info msg="Making HTTP request" url=https://example.com:8443/example/ host=
ts=2023-01-24T11:42:50.749660958Z caller=client.go:252 module=https_ipv4_status_200 target=http://example.com:8080/example level=info msg="Address does not match first address, not sending TLS ServerName" first=192.0.2.1:8080 address=example.com:8443
ts=2023-01-24T11:42:50.793033995Z caller=main.go:144 module=https_ipv4_status_200 target=http://example.com:8080/example level=info msg="Received HTTP response" status_code=200
ts=2023-01-24T11:42:50.793385379Z caller=main.go:144 module=https_ipv4_status_200 target=http://example.com:8080/example level=info msg="Response timings for roundtrip" roundtrip=0 start=2023-01-24T11:42:50.745301923Z dnsDone=2023-01-24T11:42:50.745301923Z connectDone=2023-01-24T11:42:50.745921921Z gotConn=2023-01-24T11:42:50.746004085Z responseStart=2023-01-24T11:42:50.747236065Z tlsStart=0001-01-01T00:00:00Z tlsDone=0001-01-01T00:00:00Z end=0001-01-01T00:00:00Z
ts=2023-01-24T11:42:50.793762799Z caller=main.go:144 module=https_ipv4_status_200 target=http://example.com:8080/example level=info msg="Response timings for roundtrip" roundtrip=1 start=2023-01-24T11:42:50.747851173Z dnsDone=2023-01-24T11:42:50.747851173Z connectDone=2023-01-24T11:42:50.748131529Z gotConn=2023-01-24T11:42:50.748176015Z responseStart=2023-01-24T11:42:50.74918516Z tlsStart=0001-01-01T00:00:00Z tlsDone=0001-01-01T00:00:00Z end=0001-01-01T00:00:00Z
ts=2023-01-24T11:42:50.793915312Z caller=main.go:144 module=https_ipv4_status_200 target=http://example.com:8080/example level=info msg="Response timings for roundtrip" roundtrip=2 start=2023-01-24T11:42:50.749833171Z dnsDone=2023-01-24T11:42:50.759600218Z connectDone=2023-01-24T11:42:50.760130571Z gotConn=2023-01-24T11:42:50.775857709Z responseStart=2023-01-24T11:42:50.792930037Z tlsStart=2023-01-24T11:42:50.760196377Z tlsDone=2023-01-24T11:42:50.775832Z end=2023-01-24T11:42:50.793345843Z
ts=2023-01-24T11:42:50.794118524Z caller=main.go:351 module=https_ipv4_status_200 target=http://example.com:8080/example level=info msg="Probe succeeded" duration_seconds=0.061243372

Also relevant, the output from tcpdump -i eth0 udp port 53:

11:42:50.734383 IP [our host].57761 > [DNS server]: 8334+ A? example.com. (39)
11:42:50.744439 IP [DNS server] > [our host].57761: 8334 1/0/0 A 192.0.2.1 (55)
11:42:50.750098 IP [our host].44199 > [DNS server]: 46571+ AAAA? example.com. (39)
11:42:50.750324 IP [our host.50250 > [DNS server]: 21593+ A? example.com. (39)
11:42:50.759275 IP [DNS server] > [our host].50250: 21593 1/0/0 A 192.0.21 (55)
11:42:50.759382 IP [DNS server] > [our host].44199: 46571 0/1/0 (123)

This shows the first DNS query, correctly querying only for A, as well as the second query querying for both A and AAAA. (Also the AAAA response is missing which introduces the ~5s delay of the DNS timeout.)

What did you expect to see?

Only A queries are made, even for redirects to different URLs.

What did you see instead?

See tcpdump output above. After the redirect another DNS query is made for both A and AAAA records.

[hashworks]

Works for me, can't reproduce. Uses IPv6 by default and IPv4 with preferred_ip_protocol: ip4.

[jgosmann]

@hashworks Did you test against an URL that does a redirect? It does work for the first URL, but the DNS query for the redirect URL is made for A and AAAA record even though only A should be queried.. Also, it's probably best to verify this from TCP dumps, because otherwise it becomes only noticeable with the ~5s delay in combination with some firewall (or similar) that drops the AAAA response. Without that everything will likely seem to work fine.

[hashworks]

Ah, never mind. Completely missed that redirects cause this. My bad!