How to add the probe when using the blackbox-exporter ?

[xuanyuanaosheng]

Hello, all, I have use the blackbox-exporter to do some probe: https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/blackbox-exporter.md

The probe yml is:

apiVersion: monitoring.coreos.com/v1
kind: Probe
metadata:
  name: blackbox-exporter
  namespace: monitoring-system
spec:
  jobName: http-get
  interval: 60s
  module: http_2xx
  prober:
    url: blackbox-exporter.monitoring-system.svc:19115
    scheme: http
    path: /probe
  targets:
    staticConfig:
      static:
      - https://baidu.com
      - http://qq.com

The serviceMonitor yml is

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app.kubernetes.io/component: exporter
    app.kubernetes.io/name: blackbox-exporter
    app.kubernetes.io/part-of: kube-prometheus
    app.kubernetes.io/version: 0.18.0
  name: blackbox-exporter
  namespace: monitoring-system
spec:
  endpoints:
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    interval: 30s
    path: /metrics
    port: https
    scheme: https
    tlsConfig:
      insecureSkipVerify: true
  selector:
    matchLabels:
      app.kubernetes.io/component: exporter
      app.kubernetes.io/name: blackbox-exporter
      app.kubernetes.io/part-of: kube-prometheus

The service yml is

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/component: exporter
    app.kubernetes.io/name: blackbox-exporter
    app.kubernetes.io/part-of: kube-prometheus
    app.kubernetes.io/version: 0.18.0
  name: blackbox-exporter
  namespace: monitoring-system 
spec:
  ports:
  - name: https
    port: 9115
    targetPort: https
  - name: probe
    port: 19115
    targetPort: http
  selector:
    app.kubernetes.io/component: exporter
    app.kubernetes.io/name: blackbox-exporter
    app.kubernetes.io/part-of: kube-prometheus

---

The service has two ports:

  ports:
  - name: https
    port: 9115
    targetPort: https
  - name: probe
    port: 19115
    targetPort: http

But the service monitor has only one (9115) to scrape:

  endpoints:
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    interval: 30s
    path: /metrics
    port: https
    scheme: https
    tlsConfig:
      insecureSkipVerify: true

The problem is I can not get the probe metrics like probe_success ? Does it have error in configuration?

I found that the name: probe (port: 19115) has no scrape, How to repaire this? whether to add a service monitor called probe?

[DesistDaydream]

I have a similar problem, but the difference is that I use kube-prometheus-stack Chart.

What happened? I created a Probe custom resource,but the Operator did not discovery it

Description I think the Operator should be able to correctly discover Probe CRs.

There is no error log in Operator and Prometheus. Except Probe, ServiceMonitor and PorMonitor CRs can be dicover normally

Environment Version:k8s-1.19.2 kube-prometheus-stack:15.2.0

---

Additional explanation： Although blackbox-exporter Charts can be configured by setting the value of the .serviceMonitor.targets field, it is not as convenient as using Probe CRD

---

RESOLVE This IS my problem.I did not notice the prometheus.prometheusspec.probeSelectorNilUsesHelmValues field.

[JonasJW]

Any updates on this?

[paulfantom]

blackbox-exporter deployed via kube-prometheus is using two different endpoints for two different things. On port named https we have blackbox exporter internal metrics and on probe port we have the /probe API allowing to probe other services. To use Probe CR you need to ensure that:

1. Prometheus CR has correct selectors set - Example
2. url in Probe is correctly set to use probe port from blackbox-exporter Serevice - Example
3. Probe definition is correct - Example

[paulfantom]

I am marking this as help wanted in case anyone would like to contribute short documentation on the topic. Such a doc would be much appreciated!

[JonasJW]

@paulfantom Thanks for the help!

I'm still having trouble with my probe resource. I have set the correct selectors for my prometheus resource, when I check kubectl get prometheus/prometheus-kube-prometheus-prometheus -o yaml, I can see it has probeNamespaceSelector: {} and probeSelector: {} set. My probe resource looks like this:

apiVersion: monitoring.coreos.com/v1
kind: Probe

metadata:
  name: probe-demo
  labels:
    app: kube-prometheus-stack
    chart: kube-prometheus-stack-10.1.0
    heritage: Helm
    release: prometheus

spec:
  prober:
    url: blackbox-exporter-prometheus-blackbox-exporter.svc:9115
  module: http_2xx
  targets:
    staticConfig:
      static:
        - 'https://demo.do.prometheus.io'
        - "https://google.com"
      labels:
        environment: prometheus.io

The probe url is the url of the blackbox service.

Am I missing something here?

[DesistDaydream]

This is the solution：

prometheus:
  prometheusSpec:
    serviceMonitorSelectorNilUsesHelmValues: false
    podMonitorSelectorNilUsesHelmValues: false
    probeSelectorNilUsesHelmValues: false

[paulfantom]

@JonasJW your config looks good. If this doesn't work, you need to check your RBAC rules.

[macbash]

I got the same issue, i have incl. label which is mentioned prometheus resource same in probe. its working for me.

[ritaCanavarro]

Hi @xuanyuanaosheng, I have recently written an article regarding Blackbox exporter and Kubernetes: https://medium.com/cloud-native-daily/blackbox-exporter-to-probe-or-not-to-probe-57a7a495534b, which might help you. 😊