Showing on Service Discovery but not on the Targets

[joaogl]

Hey,

I believe this is more of a miss use from my side then a bug, but at this point I've tried all documented things online and I still can't figure it out... My services are showing in the Service Discovery on prometheus but not on the targets and I can't seem to figure out what's wrong here.. At this point I don't even know what to look at to track down the issue so any help is super welcomed.

I have the helm chart installed in a namespace named kube-prometheus-stack latest version. It is a NestJS app (NodeJS) that has a /metrics endpoint exposing metrics using this plugin: https://github.com/willsoto/nestjs-prometheus

App:

apiVersion: v1
kind: Pod
metadata:
  name: be-release-default-gateway-555c6cc47f-wxzgq
  namespace: default
  labels:
    app.kubernetes.io/instance: be-system-release-default-gateway
    app.kubernetes.io/name: backend-system-gateway
spec:
  containers:
    - name: backend-system-gateway
      image: backend-system-gateway
      ports:
        - name: http
          containerPort: 8080
          protocol: TCP
      env:
        - name: NODE_ENV
          value: staging

Service:

apiVersion: v1
kind: Service
metadata:
  name: be-release-default-gateway-s
  namespace: default
  labels:
    app.kubernetes.io/component: metrics
    app.kubernetes.io/instance: be-system-release-default
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: backend-system-gateway
    app.kubernetes.io/version: 1.0.0
    helm.sh/chart: backend-system-0.1.0
    prometheus.io/path: metrics
    prometheus.io/port: '8080'
    prometheus.io/scrape: 'true'
  annotations:
    prometheus.io/path: metrics
    prometheus.io/port: '8080'
    prometheus.io/scrape: 'true'
spec:
  ports:
    - name: web-http
      protocol: TCP
      port: 8080
      targetPort: http
  selector:
    app.kubernetes.io/instance: be-system-release-default-gateway
    app.kubernetes.io/name: backend-system-gateway
  type: ClusterIP

ServiceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  annotations:
    prometheus.io/path: metrics
    prometheus.io/port: '8080'
    prometheus.io/scrape: 'true'
  labels:
    app.kubernetes.io/instance: be-system-release-default
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/version: 1.0.0
    helm.sh/chart: backend-system-0.1.0
    prometheus.io/path: metrics
    prometheus.io/port: '8080'
    prometheus.io/scrape: 'true'
  name: be-release-default-gateway-metrics
  namespace: default
spec:
  endpoints:
    - interval: 30s
      port: web-http
  namespaceSelector:
    matchNames:
      - default
  selector:
    matchLabels:
      app.kubernetes.io/component: metrics
      app.kubernetes.io/instance: be-system-release-default-gateway
      app.kubernetes.io/name: backend-system-gateway

Automatically generated prometheus config:

- job_name: serviceMonitor/default/be-release-default-gateway-metrics/0
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  follow_redirects: true
  enable_http2: true
  relabel_configs:
  - source_labels: [job]
    separator: ;
    regex: (.*)
    target_label: __tmp_prometheus_job_name
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_component, __meta_kubernetes_service_labelpresent_app_kubernetes_io_component]
    separator: ;
    regex: (metrics);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_instance, __meta_kubernetes_service_labelpresent_app_kubernetes_io_instance]
    separator: ;
    regex: (be-system-release-default-gateway);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name, __meta_kubernetes_service_labelpresent_app_kubernetes_io_name]
    separator: ;
    regex: (backend-system-gateway);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: web-http
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_container_name]
    separator: ;
    regex: (.*)
    target_label: container
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: web-http
    action: replace
  - source_labels: [__address__]
    separator: ;
    regex: (.*)
    modulus: 1
    target_label: __tmp_hash
    replacement: $1
    action: hashmod
  - source_labels: [__tmp_hash]
    separator: ;
    regex: "0"
    replacement: $1
    action: keep
  kubernetes_sd_configs:
  - role: endpoints
    kubeconfig_file: ""
    follow_redirects: true
    enable_http2: true
    namespaces:
      own_namespace: false
      names:
      - default

What do I need to look at next to try and figure out what's wrong?

Thanks in advance!

[joaogl]

As a test I just deployed everything here on local using minikube and worked nicely... So I'm guessing its something to do with RBAC... I'll keep on digging

[druanoor]

I'm having the same issue, did you managed to solve it?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[stale[bot]]

This issue is being automatically closed due to inactivity.