multi target pattern scrap - cannot init blackbox exporter address from K8S meta

[sebv004]

Host operating system: output of uname -a

blackbox_exporter version: output of blackbox_exporter --version

What is the blackbox.yml module config.

modules:
  http_2xx:
    http:
      follow_redirects: true
      preferred_ip_protocol: ip4
      valid_http_versions:
      - HTTP/1.1
      - HTTP/2.0
    prober: http
    timeout: 5s
  icmp:
    icmp:
      preferred_ip_protocol: ip4
    prober: icmp
    timeout: 5s

What is the prometheus.yml scrape config.

Based on this simple example :

scrape_configs:
  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [http_2xx]  # Look for a HTTP 200 response.
    static_configs:
      - targets:
        - http://prometheus.io    # Target to probe with http.
        - https://prometheus.io   # Target to probe with https.
        - http://example.com:8080 # Target to probe with http on port 8080.
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9115  # The blackbox exporter's real hostname:port.

I'm trying to replace the final value

• target_label: address replacement: 127.0.0.1:9115 # The blackbox exporter's real hostname:port. by a way to get my pod ip ( which is not always the same if I recreate the pod ...)
• action: replace source_labels: [__meta_kubernetes_pod_name,__meta_kubernetes_pod_container_port_number] separator: ":" target_label: address

Here is my complete scrap:

- job_name:  blackbox-icmp1
      kubernetes_sd_configs:
      - namespaces:
          names:
        role: pod
        selectors:
        - label: app.kubernetes.io/instance=blackbox-exporter
          role: pod    
      metrics_path: /probe
      params:
        module: [icmp]
      static_configs:
        - targets:
          - 10.129.100.213
          - 10.129.100.214
          - 10.129.100.215
      relabel_configs:
        - source_labels: [__address__]
          target_label: __param_target
        - source_labels: [__param_target]
          target_label: instance
        - action: replace
          source_labels: [__meta_kubernetes_pod_ip,__meta_kubernetes_pod_container_port_number]
          separator: ":"
          target_label: __address__

What logging output did you get from adding &debug=true to the probe URL?

What did you do that produced an error?

The endpoints are not formatted as expected.

What did you expect to see?

I guess, I should be able to get some endpoints like: http://my_pod_ip:9115/probe?module=icmp&target=10.129.100.213 http://my_pod_ip:9115/probe?module=icmp&target=10.129.100.214 http://my_pod_ip:9115/probe?module=icmp&target=10.129.100.215

What did you see instead?

It's a completely mess , cannot figure out whats going on ... There's one entry with my pod ip(10.233.65.239) but the target is the same ip (should be 10.129.100.21X): http://10.233.65.239:9115/probe?module=icmp&target=10.233.65.239:9115 it s looping on itself! The other endpoints are not well formatted.