[OTel to Prometheus] Streamline Prometheus metric exploration by enabling promotion of resource attributes
Proposal: Streamline Prometheus Metric Exploration by Promoting Resource Attributes
This proposal aims to simplify how OpenTelemetry metrics are explored in Prometheus by allowing key OpenTelemetry resource attributes to be promoted directly as Prometheus metric labels.
What's the Problem?
Currently, valuable OpenTelemetry resource attributes like k8s.cluster.name, deployment.environment.name, or service.version are stored exclusively within the target_info metric when converting OTLP metrics to Prometheus.
This design forces users to perform a PromQL join operation whenever they want to filter or group metric series by these common contextual metadata. This makes everyday use cases, such as analyzing metrics per environment, Kubernetes cluster, or service version, unnecessarily complex.
This complexity contrasts sharply with traditional Prometheus instrumentations, where such contextual metadata is typically included directly as metric labels, making filtering and grouping straightforward.
Consider the difference:
| Scenario | OpenTelemetry (Current) | Prometheus (Native) |
|---|---|---|
| Metric Example |
|
|
| Filter Query |
|
|
As you can see, filtering on resource attributes with current OTel metrics in Prometheus requires significantly more complex PromQL.
Proposed Enhancement
We should enable the promotion of useful OpenTelemetry resource attributes as Prometheus metric labels during the OTLP-to-Prometheus conversion. This would eliminate the need for complex PromQL joins, allowing users to filter and group metrics by these attributes directly and intuitively.
This feature isn't new; the Prometheus OTLP endpoint already supports this exact functionality via its promote_resource_attributes configuration option (docs). The OpenTelemetry Demo actively uses it to enhance metric usability (here).
Bringing this capability into the OpenTelemetry specification would standardize this beneficial behavior, ensuring a more consistent and user-friendly experience across all OpenTelemetry-to-Prometheus integrations.
Why is this important?
Implementing this change will:
- Simplify PromQL queries: Users can filter and group metrics directly, without complex joins.
- Improve discoverability: Common operational contexts (environment, K8s cluster, service version) become first-class citizens in Prometheus queries.
- Enhance parity with native Prometheus: Aligning OTel metric usability closer to native Prometheus instrumentations.
Tip: React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.
Existing configuration concepts related to this problem.
@ArthurSens @aknuds1 do you know if the question of unifying otl2prometheus configuration parameters across OTel SDK exporters and the Prometheus OTLP receiver has been discussed?
OpenTelemetry Specification / Metrics / SDK Exporters / Prometheus
https://github.com/open-telemetry/opentelemetry-specification/blob/v1.46.0/specification/metrics/sdk_exporters/prometheus.md#configuration
A Prometheus Exporter MAY offer configuration to add resource attributes as metric attributes. By default, it MUST NOT add any resource attributes as metric attributes. The configuration SHOULD allow the user to select which resource attributes to copy (e.g. include / exclude or regular expression based). Copied Resource attributes MUST NOT be excluded from the target info metric. The option MAY be named with_resource_constant_labels.
Prometheus / OpenTelemetry / OTLP Receiver
https://prometheus.io/docs/guides/opentelemetry/#promoting-resource-attributes
Promoting resource attributes Based on experience and conversations with our community, we've found that out of all the commonly seen resource attributes, there are certain worth attaching to all your OTLP metrics.
By default, Prometheus won't be promoting any attributes. If you'd like to promote any of them, you can do so in this section of the Prometheus configuration file. The following snippet shares the best practice set of attributes to promote:...
Interesting comparison @cyrille-leclerc, I also noticed that we offer a different functionality in OTel-Collector-Contrib's Prometheus exporter[1] and Prometheus Remote Write exporter[2]
The existing feature doesn't follow any of the recommended names for the configuration option, and it is a "all or nothing" strategy; you can't choose which resource attributes to promote.
I appreciate the effort of keeping the spec consistent with Prometheus' existing configuration, so maybe we can update the spec by changing the recommended configuration from with_resource_constant_labels to promote_resource_attributes?
I believe we're good to implement this functionality in both exporters in contrib as well, we just need to deprecate the old configuration option(resource_to_telemetry_conversion) through feature flags
Ok, Cyrille and I talked offline, and there's a consistency problem between Prometheus and OTel configuration options.
OTel accepts wildcards to include/exclude, while Prometheus doesn't. We'll need to figure that out first