Export Resource Metadata Labels

[raskasa]

Background

My company has an internal monitoring/alerting solution that serves both our on-premise and cloud needs; it's backed by Prometheus and Grafana. We get Cloud Monitoring (née Stackdriver) metrics into our internal solution using this exporter.

I'm currently building some Grafana graphs for Dataflow jobs.

• Metric types under the dataflow.googleapis.com namespace can be filtered down to a specific Dataflow job using the job_name label since this label's value includes the Dataflow job name. (Technically, the job_name label is attached to the dataflow_job monitored resource type).
• Metric types under the compute.googleapis.com namespace can be filtered down to a specific Dataflow job using the instance_name label since this label's value includes the Dataflow job name.
• Metric types under the agent.googleapis.com namespace CANNOT be filtered down to a specific Dataflow job. None of the labels include the Dataflow job name. The closest label that gives me what I need is the instance_id label that comes attached to the gce_instance monitored resource type. instance_id is numeric and there's no way to know what Dataflow job it belongs to.

Because I can't filter down to a specific Dataflow job with the agent.googleapis.com metric types, I can't build a complete Grafana dashboard that visualizes all the metrics I care about. Because of this, I still have to go back to using a combination of Dataflow's Job Metrics UI and Cloud Monitoring's Metrics Explorer UI just to visualize those metrics alongside others. I'm taking on the overhead of context-switching between 3 different monitoring tools. This slows down any analysis work that I want to do.

Request

As I understand it, in addition to the labels that are attached to metric types and monitored resource types, Cloud Monitoring also attaches resource metadata labels to the monitored resource types. Specifically, the metadata.system_labels.name label includes the GCE instance name which also includes the Dataflow job name. I would be able to use this label to filter the agent.googleapis.com metric types down to a specific Dataflow job.

The Stackdriver exporter doesn't seem to export these resource metadata labels though.

Is there a way we can start exporting these labels by default? Or, add a configuration that allows users to enable collection of these extra labels?

References:

• LOC where this feature might need to be implemented
• These issues look related, but not exactly the same (let me know if I'm wrong though): https://github.com/prometheus-community/stackdriver_exporter/issues/78, https://github.com/grafana/grafana/issues/18750, https://github.com/grafana/grafana/issues/18887

Happy to be corrected on anything stated here.

[xbglowx]

It would be nice to have something similar to aws_resource_info from the cloudwatch-exporter. That has allowed me to create custom tags on the aws resources themselves, i.e. owner, so that I can come up with a generic alerting rule and pass it along to alertmanager for proper routing.

From what I have seen currently, in order to do something like that with stackdriver-exporter, one would need to create multiple alerting rules for each resource, that hardcodes the needed labels?