Prometheus exporter handles instrumentation scope and prevents collisions

[dashpole]

trafficstars

Now that https://github.com/open-telemetry/opentelemetry-specification/pull/2703 is released, we can update the prometheus exporter to make use of OpenTelemetry scope, particularly for reducing collisions between metrics with the same name.

Describe the solution you'd like

There are a few components:

• Add otel_scope_name and otel_scope_version attributes to all metrics based on the instrumentation scope name and version.
• Add an otel_scope_info metric following https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/compatibility/prometheus_and_openmetrics.md#instrumentation-scope-1.
• Prevent collisions between TYPE, HELP, and UNIT comments following https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/compatibility/prometheus_and_openmetrics.md#metric-metadata-1

Note that the exporter SHOULD do the above by default, but may allow disabling the behavior.

Also, note the consideration described in https://github.com/open-telemetry/opentelemetry-python/issues/3072#issuecomment-1335842118 around grouping metrics from different scopes together in the exposition.

[mothershipper]

Not sure if this is related or not, but we're hitting an issue with the prometheus exporter and the http instrumentation. I added some debug logging and it looks like the histograms created by the instrumentation-http get created twice?

This results in a serialization error where TYPE, HELP, UNIT get duplicated -- only reason why I think it might not be the same issue as above is that these histograms are created under the same scope:

Debug code:

   for (let metric of metrics.scopeMetrics) {
     console.log(`scope: ${metric.name}`)
      for (const data of metric.metrics) {
        console.log(data.descriptor)
    }

prints:

scope: @opentelemetry/instrumentation-http
{
  name: 'http.server.duration',
  type: 'HISTOGRAM',
  description: 'measures the duration of the inbound HTTP requests',
  unit: 'ms',
  valueType: 1
}
{
  name: 'http.server.duration',
  type: 'HISTOGRAM',
  description: 'measures the duration of the inbound HTTP requests',
  unit: 'ms',
  valueType: 1
}
{
  name: 'http.client.duration',
  type: 'HISTOGRAM',
  description: 'measures the duration of the outbound HTTP requests',
  unit: 'ms',
  valueType: 1
}
{
  name: 'http.client.duration',
  type: 'HISTOGRAM',
  description: 'measures the duration of the outbound HTTP requests',
  unit: 'ms',
  valueType: 1
}

And serializes to:

# HELP http_server_duration measures the duration of the inbound HTTP requests
# UNIT http_server_duration ms
# TYPE http_server_duration histogram
# HELP http_server_duration measures the duration of the inbound HTTP requests
# UNIT http_server_duration ms
# TYPE http_server_duration histogram
# HELP http_client_duration measures the duration of the outbound HTTP requests
# UNIT http_client_duration ms
# TYPE http_client_duration histogram
# HELP http_client_duration measures the duration of the outbound HTTP requests
# UNIT http_client_duration ms
# TYPE http_client_duration histogram

[dashpole]

To me that looks like a related, but different issue. One reason why comments may be duplicated is because the same metric exists in multiple scopes, which this issue was opened to address.

It looks like you've encountered a bug in which each unique set of labels for a metric produces a new set of HELP, UNIT, and TYPE comments, which it should not do.

[mothershipper]

I've narrowed it down to the autoloader -- the Instrumentation constructor inits the metrics first, then the call to registerInstrumentation passes in the same meterProvider to enableInstrumentations, triggering the instrumentation to re-create the metrics against the original meter/meterProvider.

Would you like me to create a separate issue?

[dashpole]

I would prefer a separate issue, thanks!

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days.

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days.

[github-actions[bot]]

This issue was closed because it has been stale for 14 days with no activity.

[dashpole]

I would appreciate if this was reopened. It is still relevant.