Prometheus exporter resource attributes

[jack-berg]

The spec says the prometheus exporter SHOULD convert resource attributes to the target info metric family.

We don't currently do this, but we should.

[anuraaga]

We can probably start with populating job / instance as described

https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/data-model.md#resource-attributes-1

Other resource attributes should likely be opt-in on an attribute level or else both cardinality and size will blow up most Prometheus instances.

[jack-berg]

@anuraaga its a little unclear, but my read of that section of the spec is that the language on job and instance labels for remote write and federated pull exporter situations, such as the prometheus exporter in the collector which exposes metrics for many services and the prometheus remote write exporter for the collector. Since the SDKs only export from a single resource, that language doesn't apply.

This section provides additional context, indicating that when converting to the in memory otlp format, the collector prometheus receiver should add service.name and service.instance.id based on details from the job configuration.

My conclusion is that the only thing an SDK prometheus exporter needs to do is expose resource attributes in the target info metric family, as described here.

[anuraaga]

Ah ok, then we'd only convert the resource attributes, but I guess we need to know which are opt in or not. service seems like it is on by default to allow it to be used in the scrape config as described but process and others probably not.

[jack-berg]

Yeah the collector scrapers behavior is weird at the moment. At the current collector contrib head (with unpublished changes that parse target info), if I scrape the prometheus endpoint exposed by the sdk with the changes I made in #4569 with the following setup:

    SdkMeterProvider meterProvider = SdkMeterProvider.builder()
        .registerMetricReader(PrometheusHttpServer.create())
        .setResource(Resource.create(Attributes.builder()
            .put("service.name", "sn")
            .put("service.instance.id", "sid")
            .build()))
        .build();

    meterProvider.get("meter").counterBuilder("my-counter").build()
        .add(3, Attributes.builder().put("foo", "bar").build());

We get resource attributes as follows:

2022-06-29T18:42:47.897-0500    DEBUG   loggingexporter/logging_exporter.go:67  ResourceMetrics #0
Resource SchemaURL: 
Resource labels:
     -> service.name: STRING(app)
     -> service.instance.id: STRING(localhost:9464)
     -> net.host.port: STRING(9464)
     -> http.scheme: STRING(http)
     -> service_instance_id: STRING(sid)
     -> service_name: STRING(sn)

• service.name, service.instance.id, net.host.port, http.scheme are derived from the scrape config, as described in the spec
• net.host.name isn't available, which makes sense because its semi redundant with service.instance.id, but then by the same logic net.host.port shouldn't be included 🤷
• service_name and service_instance_id are obtained from target info, and left in their snake_case_format as required by the spec, but which is rather odd that they're not merged with service.name and service.instance.id

[jack-berg]

After chatting with @anuraaga, we think the spec needs to clarify its language. Opened this spec issue to track.

[jack-berg]

See this comment for an update on this issue.