Prometheus doesn't scrape the /metrics

[rolfberkenbosch]

Hi All,

Prometheus doesn't scrape the config from vernemq, i have the following helm config:

          image:
            repository: vernemq/vernemq
            tag: 1.10.2-1-alpine
          replicaCount: 1
          serviceMonitor:
            create: true
            enabled: true
          service:
            loadBalancerIP: 192.168.1.200
            type: LoadBalancer
            ws:
              enabled: true
          nodeSelector:
            kubernetes.io/arch: amd64
          persistentVolume:
            enabled: true
            size: 10Gi
            storageClass: "nfs-client"
          ingress:
            enabled: true
            host:
              - vernemq.domainname.nl
            annotations: 
              kubernetes.io/ingress.class: traefik
          statefulset:
            podAnnotations:
              prometheus.io/scrape: "true"
              prometheus.io/port: "8888"
          additionalEnv:
            - name: DOCKER_VERNEMQ_ACCEPT_EULA
              value: "yes"
            - name: DOCKER_VERNEMQ_ALLOW_ANONYMOUS
              value: "on"

kubectl get all -n ha
NAME                                        READY   STATUS    RESTARTS   AGE
pod/vernemq-0                               1/1     Running   0          20m
pod/svclb-vernemq-rh2dx                     2/2     Running   0          6m15s
pod/svclb-vernemq-7kqzq                     2/2     Running   0          3m35s
pod/svclb-vernemq-z5ht2                     2/2     Running   0          3m25s
pod/svclb-vernemq-8zk45                     2/2     Running   0          3m10s

NAME                           TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                         AGE
service/vernemq-headless       ClusterIP      None            <none>          4369/TCP,8888/TCP               70m
service/vernemq                LoadBalancer   10.43.52.31     192.168.1.203   1883:30162/TCP,8080:31046/TCP   70m

NAME                           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/svclb-vernemq   4         4         4       4            4           <none>          70m

NAME                       READY   AGE
statefulset.apps/vernemq   1/1     70m

It looks like that everything is running. I'm wondering is the headless service doesn't have an ClusterIP. Is that not needed for prometheus for scraping the /metrics ?

[ioolkos]

@rolfberkenbosch Hm, not sure what's the issue here. But Prometheus certainly needs to be able to contact the VerneMQ nodes to be able to scrape configs (as configured in the Prometheus scrape config).

[rolfberkenbosch]

@ioolkos So it is no issue that the service/vernemq-headless has no ClusterIP address ?

[ioolkos]

@rolfberkenbosch I don't know ;)

[a7i]

@rolfberkenbosch Headless services (by design) do not get an IP address because they do not perform any proxying or load balancing. I am testing this in my cluster right now and will report back with my findings.

https://kubernetes.io/docs/concepts/services-networking/service/#headless-services "For headless Services, a cluster IP is not allocated, kube-proxy does not handle these Services, and there is no load balancing or proxying done by the platform for them"