Honor labels option

[danielkucera]

Hello,

I am trying to scrape pushgateway using PodMonitoring object. In general it works but it does relabel job and instance labels.

To prevent relabeling, one can specify honor_labels: true in scrape config but this is currently not possible with PodMonitoring object.

Can you please add an option to enable configuration of honor_labels?

[lyanco]

Hey Daniel, could you talk a little about your use case for not relabeling the job and instance labels?

e.g. what labels do you want, and what labels are you getting from GMP?

[danielkucera]

Hi Lee,

I want to ingest metrics from external servers. For this purpose I have deployed pushgateway and I am pushing metrics from node_exporter as described here

When I use PodMonitoring for that pushgateway,

apiVersion: monitoring.googleapis.com/v1alpha1
kind: PodMonitoring
metadata:
  name: pushgateway
  namespace: gmp-frontend
spec:
  selector:
    matchLabels:
      app: pushgateway
  endpoints:
  - port: http
    interval: 60s
    path: /metrics

everything works correctly but the original instance and job labels are changed to external_instance and external_job and the standard dashboard no longer works.

I would prefer having the option to keep the original labels rather than manually modifying all dashboards.

[lyanco]

We are likely to implement this by creating a RawPodMonitoring resource that honors all labels set on the metric. Stay posted :-)

[danielkucera]

Happy to hear that! I'll follow closely :)

[Migueljfs]

@lyanco Any news on this? I've created a deployment for kube-state-metrics but the metrics that come from it have "exported_namespace" and other "exported_XXXX" labels that mess up my rules and dashboard configurations.

[lyanco]

No progress on this. On your specific note - have you looked at our copy-pastable config for ksm? It should avoid triggering those relabeling rules: https://cloud.google.com/stackdriver/docs/managed-prometheus/exporters#expandable-1

[pintohutch]

Hi @danielkucera, I believe you mean exported_instance and exported_job, correct?

@Migueljfs, you should be able to use our recommended ClusterPodMonitoring for proper relabeling of KSM metrics.

[Migueljfs]

Hey @pintohutch @lyanco

Indeed those are the labels. After replacing my kube-state-metrics with the one in the link given by @lyanco , I no longer see those exported labels, it's now properly labeled.

Thank you!

[danielkucera]

Yes, exported_instance and exported_job.