blackbox_exporter icon indicating copy to clipboard operation
blackbox_exporter copied to clipboard

Custom CA cert

Open downright-ux opened this issue 1 year ago • 2 comments

I 'm trying to configure blackbox to work with our custom CA . I see issue in the logs .

alertmanager       | ts=2022-08-17T17:41:34.355Z caller=notify.go:732 level=warn component=dispatcher receiver=team-1 integration=email[0] msg="Notify attempt failed, will retry later" attempts=1 err="send STARTTLS command: x509: certificate signed by unknown authority"

My configuration for blackbox.yaml

modules:
  http_2xx:
    prober: http
  http_post_2xx:
    prober: http
    http:
      method: POST
      tls_config:
        ca_file: "/opt/bitnami/blackbox-exporter/cacert.crt"
  http_custom_ca:
    prober: http
    http:
      method: GET
      tls_config:
        ca_file: "/opt/bitnami/blackbox-exporter/cacert.crt"  
  tcp_connect:
    prober: tcp
  pop3s_banner:
    prober: tcp
    tcp:
      query_response:
      - expect: "^+OK"
      tls: true
      tls_config:
        insecure_skip_verify: false
  grpc:
    prober: grpc
    grpc:
      tls: true
      preferred_ip_protocol: "ip4"
  grpc_plain:
    prober: grpc
    grpc:
      tls: false
      service: "service1"
  ssh_banner:
    prober: tcp
    tcp:
      query_response:
      - expect: "^SSH-2.0-"
      - send: "SSH-2.0-blackbox-ssh-check"
  irc_banner:
    prober: tcp
    tcp:
      query_response:
      - send: "NICK prober"
      - send: "USER prober prober prober :prober"
      - expect: "PING :([^ ]+)"
        send: "PONG ${1}"
      - expect: "^:[^ ]+ 001"
  icmp:
    prober: icmp
    ```
    and docker-compose looks like this
    ```
    version: '3'

services:
  blackbox-exporter:
    container_name: blackbox-exporter
    image: bitnami/blackbox-exporter
    network_mode: host
    restart: always
    ports:
      - "9115:9115"
    volumes:
      - /data/monitoring/blackbox/cacert.crt:/opt/bitnami/blackbox-exporter/cacert.crt
      - /data/monitoring/blackbox/blackbox.yaml:/opt/bitnami/blackbox-exporter/blackbox.yml
 
  alertmanager:
    image: prom/alertmanager
    container_name: alertmanager
    ports:
      - 9093:9093
    volumes:
      - ./alertmanager/:/etc/alertmanager/
    network_mode: host
    restart: always
    ```
    
    Any help here would be great. Thank you

downright-ux avatar Aug 17 '22 17:08 downright-ux

@downright-ux did you ever solve this? I have got the exact same issue.

My hosting provider did some migration and appears to have changed my Certificate Authority. Both wget and curl to my website was suddenly failing due to certificate errors. I solved that by manually downloading the CA certificate and updating certificates (sudo dpkg-reconfigure ca-certificates) on Ubuntu.

Blackbox probes started failing at the same time and I assume it's the same problem. I tried to specify the certificate file using tls_config and ca_file as you have done in your config above. But it makes no difference, still getting 403 forbidden errors with or without the tls_config.

I also tried adding insecure_skip_verify: true but no difference.

therealcodlord avatar Jan 28 '24 11:01 therealcodlord