opentelemetry-dotnet icon indicating copy to clipboard operation
opentelemetry-dotnet copied to clipboard

[bug] Cumulative Counts in Native Histogram

Open CraigKennedy opened this issue 3 weeks ago • 1 comments

Package

OpenTelemetry.Api

Package Version

Package Name Version
OpenTelemetry.Api 1.12.0
OpenTelemetry 1.12.0

Runtime Version

net8.0

Description

The counter continually increases and queries from a grafana dashboard do not reflect the actual number of requests made in a given period.

Steps to Reproduce

Here is the code where I configure the metrics collection. I'm using a sidecar container running Grafana Alloy to ship the metrics to Grafana Cloud.

            telemetryBuilder.WithMetrics(builder =>
            {
                builder
                    .SetResourceBuilder(ResourceBuilder.CreateEmpty().AddAttributes(attrs))
                    .ConfigureResource(resource => resource.AddService(applicationName, autoGenerateServiceInstanceId: false))
                    .AddAspNetCoreInstrumentation()
                    .AddApplicationMetrics()
                    .AddOtlpExporter(options =>
                    {
                        options.Endpoint = new Uri(otlpExporterEndpoint);
                        options.Protocol = otlpProtocol;
                    });

                builder.AddView(instrument => instrument.GetType().GetGenericTypeDefinition() == typeof(Histogram<>)
                    ? new Base2ExponentialBucketHistogramConfiguration()
                    : null);
            });

When I try to visualize http_server_request_duration_seconds in grafana using the histogram view with the query histogram_count(rate(http_server_request_duration_seconds{http_route="v1/coo/{part}", region="us-east-1", http_response_status_code="200", environment="prod"}[$__rate_interval])), the value only ever increases (until an instance restart).

Here is a graph of the query shown above: Image

Expected Result

I expect the query to provide actual observed requests over time.

Actual Result

The request count continually increases over time.

Additional Context

No response

Tip

React with :+1: to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.

CraigKennedy avatar Dec 03 '25 19:12 CraigKennedy

I think this is normal cumulative behavior. It's best if you consult your backend vendor (Grafana) to make sure the queries are correct. Even in prometheus (which I think is backing the UI you have shared), it only accepts cumulative, so you need to write queries to take that into account.

cijothomas avatar Dec 03 '25 23:12 cijothomas