[Feature Request] Add custom prometheus labels

[rajat404]

In event of multiple Kafka clusters pushing metrics to the same Prometheus instance, it can be hard to differentiate between metrics.

One solution is to add a custom label (say kafka_cluster_name) in all the metrics, which can be used as a unique identifier.

I'd be happy to submit a PR for this, if you're amenable to having this feature 😄

[weeco]

Personally I'd use metadata annotations for the deployment and configure prometheus so that it's being picked up at scrape time.

However I can see others using Kafka Minion outside of Kubernetes and I am not sure how much effort that would cause there. Therefore I would tend to await more feedback and accept a PR if there's interest from the community.

[ohjongsung]

yes, I need to this feature.

[ArjonBu]

This feature is also very welcome.

[weeco]

Even though I really want to incorporate community feedback I'm still hesitating to implement this. If you want a static label for all of the exported labels of KMinion the way to do this is to add the labels at scrape time by adapting your Prometheus job config. In a Kubernetes context you could configure your Prometheus job so that it would automatically attach metadata labels from the scraped container/pod.

It feels wrong to add a const label on every single Prometheus metric. Can someone explain in more detail why this should be done as part of the exporter rather than adapting the Prometheus scrape config?

[stoader]

@weeco I agree with using metadata labels supporting additional labels on metrics published by Kafka Minion should be done through the service monitor deployed by the Helm chart. That's is a fexible solution which doesn't require changing the Kafka Minion codebase.

https://github.com/cloudhut/kminion/blob/master/charts/kminion/templates/servicemonitor.yaml#L12-L20 could be modified to look like:

spec:
  selector:
    matchLabels:
      {{- include "kminion.labels" . | nindent 6}}
  endpoints:
    - port: metrics
      path: /metrics
      honorLabels: {{ .Values.serviceMonitor.honorLabels }}
      {{- if .Values.serviceMonitor.relabelings }}
      relabelings:
      {{ toYaml .Values.serviceMonitor.relabelings | nindent 4 }}

An example of relabeling config that would add the my_label from the Kafka Minion deployment to metrics as my_label:

- sourceLabels: [__meta_kubernetes_pod_label_my_label]
  separator: ;
  regex: (.*)
  targetLabel: my_label
  replacement: $1
  action: replace

[nahsi]

This is how it is done with consul:

service {
  name = "kafka"
  port = "kafka"
  tags = [ "cluster=prod", "foo=bar" ]
}

- job_name: "kafka"
    consul_sd_configs:
      - services:
          - "kafka"
    relabel_configs:
      - source_labels: ["__meta_consul_node"]
        target_label: "instance"
      - source_labels: ["__meta_consul_tags"]
        regex: ".*,([^=]+)=([^,]+),.*"
        replacement: "$2"
        target_label: "$1"

Prometheus metrics scraped from kafka would have additional labels cluster and foo.

[stoader]

@weeco I agree with using metadata labels supporting additional labels on metrics published by Kafka Minion should be done through the service monitor deployed by the Helm chart. That's is a fexible solution which doesn't require changing the Kafka Minion codebase.

https://github.com/cloudhut/kminion/blob/master/charts/kminion/templates/servicemonitor.yaml#L12-L20 could be modified to look like:

spec:
  selector:
    matchLabels:
      {{- include "kminion.labels" . | nindent 6}}
  endpoints:
    - port: metrics
      path: /metrics
      honorLabels: {{ .Values.serviceMonitor.honorLabels }}
      {{- if .Values.serviceMonitor.relabelings }}
      relabelings:
      {{ toYaml .Values.serviceMonitor.relabelings | nindent 4 }}

An example of relabeling config that would add the my_label from the Kafka Minion deployment to metrics as my_label:

- sourceLabels: [__meta_kubernetes_pod_label_my_label]
  separator: ;
  regex: (.*)
  targetLabel: my_label
  replacement: $1
  action: replace

Hi @weeco, any comment on this suggested change to allow specifying relabelings through values.yaml?

[weeco]

Hey @stoader , sorry for my late response: Yes I'm fine with this. Feel free to open a PR! Thx

[weeco]

I'll consider this as fixed since this is possible to do via the service monitor, see: https://github.com/redpanda-data/kminion/pull/122/files