opentelemetry-php icon indicating copy to clipboard operation
opentelemetry-php copied to clipboard

Published 20 hours ago • verified publisher icon open-telemetry

Counter Issue not working with example code in opentelemetry docs

Open devsekar-1992 opened this issue 6 months ago • 8 comments

❗ Metrics not pushed to Prometheus from PHP SDK (Counter not working)

We are working on a legacy CRM system, and we’re adding full observability using OpenTelemetry, Grafana Alloy, and Prometheus.

  • ✅ Traces and logs are working via Alloy and OTLP
  • ✅ Metrics using ObservableGauge are working
  • ❌ Metrics using Counter::add() are collected by Alloy, but do not show up in Prometheus

✅ Environment

Component Version / Details
PHP 8.2.28
OpenTelemetry PHP SDK 1.2.4
Alloy Latest (May 2025)
Prometheus v2.51+
Export Protocol OTLP over HTTP (x-protobuf)
Prometheus remote_write Enabled (/api/v1/write)

📜 Sample Code

require_once 'vendor/autoload.php';

use OpenTelemetry\Contrib\Otlp\MetricExporter;
use OpenTelemetry\Contrib\Otlp\OtlpHttpTransportFactory;
use OpenTelemetry\SDK\Metrics\MeterProvider;
use OpenTelemetry\SDK\Metrics\MetricReader\ExportingReader;
use OpenTelemetry\SDK\Resource\ResourceInfo;
use OpenTelemetry\SemConv\ResourceAttributes;
use OpenTelemetry\SDK\Common\Attribute\Attributes;
use OpenTelemetry\SDK\Common\Time\ClockFactory;
use OpenTelemetry\SDK\Resource\ResourceInfoFactory;
use OpenTelemetry\SDK\Common\Instrumentation\InstrumentationScopeFactory;
use OpenTelemetry\SDK\Metrics\StalenessHandler\ImmediateStalenessHandlerFactory;
use OpenTelemetry\SDK\Metrics\Exemplar\ExemplarFilter\WithSampledTraceExemplarFilter;
use OpenTelemetry\SDK\Metrics\View\CriteriaViewRegistry;

$transport = (new OtlpHttpTransportFactory())->create(
    'http://localhost:4318/v1/metrics',
    'application/x-protobuf'
);

$reader = new ExportingReader(new MetricExporter($transport), ClockFactory::getDefault());

$resource = ResourceInfo::create(Attributes::create([
    ResourceAttributes::SERVICE_NAME => 'counter-test',
]));

$provider = new MeterProvider(
    null,
    ResourceInfoFactory::emptyResource(),
    ClockFactory::getDefault(),
    Attributes::factory(),
    new InstrumentationScopeFactory(Attributes::factory()),
    [$reader],
    new CriteriaViewRegistry(),
    new WithSampledTraceExemplarFilter(),
    new ImmediateStalenessHandlerFactory(),
);

$meter = $provider->getMeter('io.opentelemetry.contrib.php');

$counter = $meter->createCounter('test.counter.total', 'ms', 'A simple counter.');
$counter->add(1, ['label' => 'test']);

$reader->collect();
$provider->forceFlush();

📦 Alloy Log Output

Alloy successfully logs this:

Metric #0
-> Name: test.counter.total
-> DataType: Sum
-> IsMonotonic: true
-> AggregationTemporality: Delta
-> Value: 1

❌ Problem

Prometheus only shows:

sample: ts=..., labels={__name__="target_info"}, value=1.0

No counter metric appears.

✅ Comparison with Python SDK

Using Python with:

meter.create_counter("test.counter.total").add(1)

✅ Metric appears in Prometheus immediately.

✅ ObservableGauge Works in PHP

If we simulate the counter like this:

$meter->createObservableGauge('test.counter.total')
    ->observe(fn($observer) => $observer->observe(1, ['label' => 'test']));

✅ It appears in Prometheus without issue.

❓ Expected Behavior

  • Counter::add() should produce a flushable metric
  • It should show up in Prometheus after collect() + forceFlush()

❓ Actual Behavior

  • Metric is collected by Alloy and logged
  • Prometheus drops or does not store the sample
  • Same metric works from Python, not from PHP

🔍 Hypothesis

  1. PHP SDK uses AggregationTemporality = Delta, while Prometheus prefers CUMULATIVE
  2. StartTimestamp may reset or be incomplete for Prometheus to stitch time series
  3. Prometheus silently drops delta counters that do not follow its expectations

✅ Alloy Configuration Used

otelcol.receiver.otlp "default" {
  http {
    endpoint = "0.0.0.0:4318"
  }
  output {
    metrics = [otelcol.processor.batch.default.input]
  }
}

otelcol.processor.batch "default" {
  output {
    metrics = [otelcol.exporter.prometheus.default.input]
  }
}

otelcol.exporter.prometheus "default" {
  forward_to = [prometheus.remote_write.default.receiver]
}

prometheus.remote_write "default" {
  endpoint {
    url = "http://localhost:9090/api/v1/write"
  }
}

devsekar-1992 avatar May 09 '25 10:05 devsekar-1992

I think we need a little bit more information about your environment to correctly assess this situation. Will you fill out the rest of the issue template when you have moment please? Referenced below for convenience:

**Describe your environment** Describe any aspect of your environment relevant to the problem, including your php version (`php -v` will tell you your current version), version numbers of installed dependencies, information about your cloud hosting provider, etc. If you're reporting a problem with a specific version of a library in this repo, please check whether the problem has been fixed on master.

**Steps to reproduce**
Describe exactly how to reproduce the error. Include a code sample if applicable.

**What is the expected behavior?**
What did you expect to see?

**What is the actual behavior?**
What did you see instead?

**Additional context**
Add any other context about the problem here.

bobstrecansky avatar May 09 '25 11:05 bobstrecansky

@bobstrecansky

I updated please help on this

devsekar-1992 avatar May 09 '25 12:05 devsekar-1992

@devsekar-1992 you said that a similar counter from python SDK worked as expected? If you export both php + python to an opentelemetry collector with debug exporter + detailed verbosity, can you see any difference between the counters?

brettmc avatar Jun 13 '25 02:06 brettmc

@brettmc Thanks for replying. we are using grafana alloy for this telemetry collector. I checked with logs for both python and php. No luck to find why its not working in php.

Even this example: Histogram also not working https://github.com/open-telemetry/opentelemetry-php/blob/main/examples/metrics/exporters/otlp_http.php https://github.com/open-telemetry/opentelemetry-php/blob/main/examples/src/ExampleMetricsGenerator.php

devsekar-1992 avatar Jun 13 '25 13:06 devsekar-1992

It would be really helpful if you could export to an opentelemetry collector and dump out the metrics as received there. We need to see how the data differs b/w php+python, not necessarily whether any errors were logged.

edit: it seems that Alloy is based on the opentelemetry collector, so perhaps adding this config will be enough to see and compare the metrics data from php vs python?

exporters:
  debug:
    verbosity: detailed

service:
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [debug]

brettmc avatar Jun 14 '25 08:06 brettmc

@brettmc

Log for php

ResourceMetrics #0
Resource SchemaURL: https://opentelemetry.io/schemas/1.30.0
Resource attributes:
     -> host.name: Str(VALOR-LP-25)
     -> host.arch: Str(x86_64)
     -> host.id: Str(6af20bfa5f254671bec36b426d997b38)
     -> os.type: Str(linux)
     -> os.description: Str(4.4.0-22621-Microsoft)
     -> os.name: Str(Linux)
     -> os.version: Str(#4391-Microsoft Fri Jan 01 08:00:00 PST 2016)
     -> process.pid: Int(20723)
     -> process.executable.path: Str(/usr/bin/php8.4)
     -> process.command: Str(tesmetrics.php)
     -> process.command_args: Slice(["tesmetrics.php"])
     -> process.owner: Str(root)
     -> process.runtime.name: Str(cli)
     -> process.runtime.version: Str(8.4.8)
     -> telemetry.sdk.name: Str(opentelemetry)
     -> telemetry.sdk.language: Str(php)
     -> telemetry.sdk.version: Str(1.2.4)
     -> service.name: Str(bar)
     -> service.version: Str(0.1)
     -> service.namespace: Str(foo)
     -> service.instance.id: Int(1)
     -> deployment.environment.name: Str(development)
ScopeMetrics #0
ScopeMetrics SchemaURL: 
InstrumentationScope demo-meter 
Metric #0
Descriptor:
     -> Name: demo_counter
     -> Description: 
     -> Unit: 
     -> DataType: Sum
     -> IsMonotonic: true
     -> AggregationTemporality: Delta
NumberDataPoints #0
Data point attributes:
     -> env: Str(dev)
StartTimestamp: 2025-06-21 07:38:11.47302314 +0000 UTC
Timestamp: 2025-06-21 07:38:11.53343844 +0000 UTC
Value: 1

Log for python

Click to open 
ResourceMetrics #0
Resource SchemaURL: 
Resource attributes:
     -> telemetry.sdk.language: Str(python)
     -> telemetry.sdk.name: Str(opentelemetry)
     -> telemetry.sdk.version: Str(1.34.1)
     -> service.name: Str(unknown_service)
ScopeMetrics #0
ScopeMetrics SchemaURL: 
InstrumentationScope example-meter 
Metric #0
Descriptor:
     -> Name: http_requests_total
     -> Description: Total number of HTTP requests
     -> Unit: 1
     -> DataType: Sum
     -> IsMonotonic: true
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> service.name: Str(demo-service)
     -> env: Str(dev)
StartTimestamp: 2025-06-21 08:08:44.0995058 +0000 UTC
Timestamp: 2025-06-21 08:08:49.0897675 +0000 UTC
Value: 3
Metric #1
Descriptor:
     -> Name: http_response_duration_seconds
     -> Description: Response duration in seconds
     -> Unit: s
     -> DataType: Histogram
     -> AggregationTemporality: Cumulative
HistogramDataPoints #0
Data point attributes:
     -> service.name: Str(demo-service)
     -> env: Str(dev)
StartTimestamp: 2025-06-21 08:08:44.099559 +0000 UTC
Timestamp: 2025-06-21 08:08:49.0897675 +0000 UTC
Count: 3
Sum: 0.450000
Min: 0.100000
Max: 0.200000
ExplicitBounds #0: 0.000000
ExplicitBounds #1: 5.000000
ExplicitBounds #2: 10.000000
ExplicitBounds #3: 25.000000
ExplicitBounds #4: 50.000000
ExplicitBounds #5: 75.000000
ExplicitBounds #6: 100.000000
ExplicitBounds #7: 250.000000
ExplicitBounds #8: 500.000000
ExplicitBounds #9: 750.000000
ExplicitBounds #10: 1000.000000
ExplicitBounds #11: 2500.000000
ExplicitBounds #12: 5000.000000
ExplicitBounds #13: 7500.000000
ExplicitBounds #14: 10000.000000
Buckets #0, Count: 0
Buckets #1, Count: 3
Buckets #2, Count: 0
Buckets #3, Count: 0
Buckets #4, Count: 0
Buckets #5, Count: 0
Buckets #6, Count: 0
Buckets #7, Count: 0
Buckets #8, Count: 0
Buckets #9, Count: 0
Buckets #10, Count: 0
Buckets #11, Count: 0
Buckets #12, Count: 0
Buckets #13, Count: 0
Buckets #14, Count: 0
Buckets #15, Count: 0

ResourceMetrics #0
Resource SchemaURL: 
Resource attributes:
     -> telemetry.sdk.language: Str(python)
     -> telemetry.sdk.name: Str(opentelemetry)
     -> telemetry.sdk.version: Str(1.34.1)
     -> service.name: Str(unknown_service)
ScopeMetrics #0
ScopeMetrics SchemaURL: 
InstrumentationScope example-meter 
Metric #0
Descriptor:
     -> Name: http_requests_total
     -> Description: Total number of HTTP requests
     -> Unit: 1
     -> DataType: Sum
     -> IsMonotonic: true
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> service.name: Str(demo-service)
     -> env: Str(dev)
StartTimestamp: 2025-06-21 08:08:44.0995058 +0000 UTC
Timestamp: 2025-06-21 08:08:54.1294906 +0000 UTC
Value: 6
Metric #1
Descriptor:
     -> Name: http_response_duration_seconds
     -> Description: Response duration in seconds
     -> Unit: s
     -> DataType: Histogram
     -> AggregationTemporality: Cumulative
HistogramDataPoints #0
Data point attributes:
     -> service.name: Str(demo-service)
     -> env: Str(dev)
StartTimestamp: 2025-06-21 08:08:44.099559 +0000 UTC
Timestamp: 2025-06-21 08:08:54.1294906 +0000 UTC
Count: 6
Sum: 1.350000
Min: 0.100000
Max: 0.350000
ExplicitBounds #0: 0.000000
ExplicitBounds #1: 5.000000
ExplicitBounds #2: 10.000000
ExplicitBounds #3: 25.000000
ExplicitBounds #4: 50.000000
ExplicitBounds #5: 75.000000
ExplicitBounds #6: 100.000000
ExplicitBounds #7: 250.000000
ExplicitBounds #8: 500.000000
ExplicitBounds #9: 750.000000
ExplicitBounds #10: 1000.000000
ExplicitBounds #11: 2500.000000
ExplicitBounds #12: 5000.000000
ExplicitBounds #13: 7500.000000
ExplicitBounds #14: 10000.000000
Buckets #0, Count: 0
Buckets #1, Count: 6
Buckets #2, Count: 0
Buckets #3, Count: 0
Buckets #4, Count: 0
Buckets #5, Count: 0
Buckets #6, Count: 0
Buckets #7, Count: 0
Buckets #8, Count: 0
Buckets #9, Count: 0
Buckets #10, Count: 0
Buckets #11, Count: 0
Buckets #12, Count: 0
Buckets #13, Count: 0
Buckets #14, Count: 0
Buckets #15, Count: 0

ResourceMetrics #0
Resource SchemaURL: 
Resource attributes:
     -> telemetry.sdk.language: Str(python)
     -> telemetry.sdk.name: Str(opentelemetry)
     -> telemetry.sdk.version: Str(1.34.1)
     -> service.name: Str(unknown_service)
ScopeMetrics #0
ScopeMetrics SchemaURL: 
InstrumentationScope example-meter 
Metric #0
Descriptor:
     -> Name: http_requests_total
     -> Description: Total number of HTTP requests
     -> Unit: 1
     -> DataType: Sum
     -> IsMonotonic: true
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> service.name: Str(demo-service)
     -> env: Str(dev)
StartTimestamp: 2025-06-21 08:08:44.0995058 +0000 UTC
Timestamp: 2025-06-21 08:08:59.1383503 +0000 UTC
Value: 8
Metric #1
Descriptor:
     -> Name: http_response_duration_seconds
     -> Description: Response duration in seconds
     -> Unit: s
     -> DataType: Histogram
     -> AggregationTemporality: Cumulative
HistogramDataPoints #0
Data point attributes:
     -> service.name: Str(demo-service)
     -> env: Str(dev)
StartTimestamp: 2025-06-21 08:08:44.099559 +0000 UTC
Timestamp: 2025-06-21 08:08:59.1383503 +0000 UTC
Count: 8
Sum: 2.200000
Min: 0.100000
Max: 0.450000
ExplicitBounds #0: 0.000000
ExplicitBounds #1: 5.000000
ExplicitBounds #2: 10.000000
ExplicitBounds #3: 25.000000
ExplicitBounds #4: 50.000000
ExplicitBounds #5: 75.000000
ExplicitBounds #6: 100.000000
ExplicitBounds #7: 250.000000
ExplicitBounds #8: 500.000000
ExplicitBounds #9: 750.000000
ExplicitBounds #10: 1000.000000
ExplicitBounds #11: 2500.000000
ExplicitBounds #12: 5000.000000
ExplicitBounds #13: 7500.000000
ExplicitBounds #14: 10000.000000
Buckets #0, Count: 0
Buckets #1, Count: 8
Buckets #2, Count: 0
Buckets #3, Count: 0
Buckets #4, Count: 0
Buckets #5, Count: 0
Buckets #6, Count: 0
Buckets #7, Count: 0
Buckets #8, Count: 0
Buckets #9, Count: 0
Buckets #10, Count: 0
Buckets #11, Count: 0
Buckets #12, Count: 0
Buckets #13, Count: 0
Buckets #14, Count: 0
Buckets #15, Count: 0

ResourceMetrics #0
Resource SchemaURL: 
Resource attributes:
     -> telemetry.sdk.language: Str(python)
     -> telemetry.sdk.name: Str(opentelemetry)
     -> telemetry.sdk.version: Str(1.34.1)
     -> service.name: Str(unknown_service)
ScopeMetrics #0
ScopeMetrics SchemaURL: 
InstrumentationScope example-meter 
Metric #0
Descriptor:
     -> Name: http_requests_total
     -> Description: Total number of HTTP requests
     -> Unit: 1
     -> DataType: Sum
     -> IsMonotonic: true
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> service.name: Str(demo-service)
     -> env: Str(dev)
StartTimestamp: 2025-06-21 08:08:44.0995058 +0000 UTC
Timestamp: 2025-06-21 08:09:04.1099795 +0000 UTC
Value: 10
Metric #1
Descriptor:
     -> Name: http_response_duration_seconds
     -> Description: Response duration in seconds
     -> Unit: s
     -> DataType: Histogram
     -> AggregationTemporality: Cumulative
HistogramDataPoints #0
Data point attributes:
     -> service.name: Str(demo-service)
     -> env: Str(dev)
StartTimestamp: 2025-06-21 08:08:44.099559 +0000 UTC
Timestamp: 2025-06-21 08:09:04.1099795 +0000 UTC
Count: 10
Sum: 3.250000
Min: 0.100000
Max: 0.550000
ExplicitBounds #0: 0.000000
ExplicitBounds #1: 5.000000
ExplicitBounds #2: 10.000000
ExplicitBounds #3: 25.000000
ExplicitBounds #4: 50.000000
ExplicitBounds #5: 75.000000
ExplicitBounds #6: 100.000000
ExplicitBounds #7: 250.000000
ExplicitBounds #8: 500.000000
ExplicitBounds #9: 750.000000
ExplicitBounds #10: 1000.000000
ExplicitBounds #11: 2500.000000
ExplicitBounds #12: 5000.000000
ExplicitBounds #13: 7500.000000
ExplicitBounds #14: 10000.000000
Buckets #0, Count: 0
Buckets #1, Count: 10
Buckets #2, Count: 0
Buckets #3, Count: 0
Buckets #4, Count: 0
Buckets #5, Count: 0
Buckets #6, Count: 0
Buckets #7, Count: 0
Buckets #8, Count: 0
Buckets #9, Count: 0
Buckets #10, Count: 0
Buckets #11, Count: 0
Buckets #12, Count: 0
Buckets #13, Count: 0
Buckets #14, Count: 0
Buckets #15, Count: 0

ResourceMetrics #0
Resource SchemaURL: 
Resource attributes:
     -> telemetry.sdk.language: Str(python)
     -> telemetry.sdk.name: Str(opentelemetry)
     -> telemetry.sdk.version: Str(1.34.1)
     -> service.name: Str(unknown_service)
ScopeMetrics #0
ScopeMetrics SchemaURL: 
InstrumentationScope example-meter 
Metric #0
Descriptor:
     -> Name: http_requests_total
     -> Description: Total number of HTTP requests
     -> Unit: 1
     -> DataType: Sum
     -> IsMonotonic: true
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> service.name: Str(demo-service)
     -> env: Str(dev)
StartTimestamp: 2025-06-21 08:10:25.2128074 +0000 UTC
Timestamp: 2025-06-21 08:10:30.2030601 +0000 UTC
Value: 3
Metric #1
Descriptor:
     -> Name: http_response_duration_seconds
     -> Description: Response duration in seconds
     -> Unit: s
     -> DataType: Histogram
     -> AggregationTemporality: Cumulative
HistogramDataPoints #0
Data point attributes:
     -> service.name: Str(demo-service)
     -> env: Str(dev)
StartTimestamp: 2025-06-21 08:10:25.2128411 +0000 UTC
Timestamp: 2025-06-21 08:10:30.2030601 +0000 UTC
Count: 3
Sum: 0.450000
Min: 0.100000
Max: 0.200000
ExplicitBounds #0: 0.000000
ExplicitBounds #1: 5.000000
ExplicitBounds #2: 10.000000
ExplicitBounds #3: 25.000000
ExplicitBounds #4: 50.000000
ExplicitBounds #5: 75.000000
ExplicitBounds #6: 100.000000
ExplicitBounds #7: 250.000000
ExplicitBounds #8: 500.000000
ExplicitBounds #9: 750.000000
ExplicitBounds #10: 1000.000000
ExplicitBounds #11: 2500.000000
ExplicitBounds #12: 5000.000000
ExplicitBounds #13: 7500.000000
ExplicitBounds #14: 10000.000000
Buckets #0, Count: 0
Buckets #1, Count: 3
Buckets #2, Count: 0
Buckets #3, Count: 0
Buckets #4, Count: 0
Buckets #5, Count: 0
Buckets #6, Count: 0
Buckets #7, Count: 0
Buckets #8, Count: 0
Buckets #9, Count: 0
Buckets #10, Count: 0
Buckets #11, Count: 0
Buckets #12, Count: 0
Buckets #13, Count: 0
Buckets #14, Count: 0
Buckets #15, Count: 0

ResourceMetrics #0
Resource SchemaURL: 
Resource attributes:
     -> telemetry.sdk.language: Str(python)
     -> telemetry.sdk.name: Str(opentelemetry)
     -> telemetry.sdk.version: Str(1.34.1)
     -> service.name: Str(unknown_service)
ScopeMetrics #0
ScopeMetrics SchemaURL: 
InstrumentationScope example-meter 
Metric #0
Descriptor:
     -> Name: http_requests_total
     -> Description: Total number of HTTP requests
     -> Unit: 1
     -> DataType: Sum
     -> IsMonotonic: true
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> service.name: Str(demo-service)
     -> env: Str(dev)
StartTimestamp: 2025-06-21 08:10:25.2128074 +0000 UTC
Timestamp: 2025-06-21 08:10:35.2216811 +0000 UTC
Value: 6
Metric #1
Descriptor:
     -> Name: http_response_duration_seconds
     -> Description: Response duration in seconds
     -> Unit: s
     -> DataType: Histogram
     -> AggregationTemporality: Cumulative
HistogramDataPoints #0
Data point attributes:
     -> service.name: Str(demo-service)
     -> env: Str(dev)
StartTimestamp: 2025-06-21 08:10:25.2128411 +0000 UTC
Timestamp: 2025-06-21 08:10:35.2216811 +0000 UTC
Count: 6
Sum: 1.350000
Min: 0.100000
Max: 0.350000
ExplicitBounds #0: 0.000000
ExplicitBounds #1: 5.000000
ExplicitBounds #2: 10.000000
ExplicitBounds #3: 25.000000
ExplicitBounds #4: 50.000000
ExplicitBounds #5: 75.000000
ExplicitBounds #6: 100.000000
ExplicitBounds #7: 250.000000
ExplicitBounds #8: 500.000000
ExplicitBounds #9: 750.000000
ExplicitBounds #10: 1000.000000
ExplicitBounds #11: 2500.000000
ExplicitBounds #12: 5000.000000
ExplicitBounds #13: 7500.000000
ExplicitBounds #14: 10000.000000
Buckets #0, Count: 0
Buckets #1, Count: 6
Buckets #2, Count: 0
Buckets #3, Count: 0
Buckets #4, Count: 0
Buckets #5, Count: 0
Buckets #6, Count: 0
Buckets #7, Count: 0
Buckets #8, Count: 0
Buckets #9, Count: 0
Buckets #10, Count: 0
Buckets #11, Count: 0
Buckets #12, Count: 0
Buckets #13, Count: 0
Buckets #14, Count: 0
Buckets #15, Count: 0

ResourceMetrics #0
Resource SchemaURL: 
Resource attributes:
     -> telemetry.sdk.language: Str(python)
     -> telemetry.sdk.name: Str(opentelemetry)
     -> telemetry.sdk.version: Str(1.34.1)
     -> service.name: Str(unknown_service)
ScopeMetrics #0
ScopeMetrics SchemaURL: 
InstrumentationScope example-meter 
Metric #0
Descriptor:
     -> Name: http_requests_total
     -> Description: Total number of HTTP requests
     -> Unit: 1
     -> DataType: Sum
     -> IsMonotonic: true
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> service.name: Str(demo-service)
     -> env: Str(dev)
StartTimestamp: 2025-06-21 08:10:25.2128074 +0000 UTC
Timestamp: 2025-06-21 08:10:40.2338063 +0000 UTC
Value: 8
Metric #1
Descriptor:
     -> Name: http_response_duration_seconds
     -> Description: Response duration in seconds
     -> Unit: s
     -> DataType: Histogram
     -> AggregationTemporality: Cumulative
HistogramDataPoints #0
Data point attributes:
     -> service.name: Str(demo-service)
     -> env: Str(dev)
StartTimestamp: 2025-06-21 08:10:25.2128411 +0000 UTC
Timestamp: 2025-06-21 08:10:40.2338063 +0000 UTC
Count: 8
Sum: 2.200000
Min: 0.100000
Max: 0.450000
ExplicitBounds #0: 0.000000
ExplicitBounds #1: 5.000000
ExplicitBounds #2: 10.000000
ExplicitBounds #3: 25.000000
ExplicitBounds #4: 50.000000
ExplicitBounds #5: 75.000000
ExplicitBounds #6: 100.000000
ExplicitBounds #7: 250.000000
ExplicitBounds #8: 500.000000
ExplicitBounds #9: 750.000000
ExplicitBounds #10: 1000.000000
ExplicitBounds #11: 2500.000000
ExplicitBounds #12: 5000.000000
ExplicitBounds #13: 7500.000000
ExplicitBounds #14: 10000.000000
Buckets #0, Count: 0
Buckets #1, Count: 8
Buckets #2, Count: 0
Buckets #3, Count: 0
Buckets #4, Count: 0
Buckets #5, Count: 0
Buckets #6, Count: 0
Buckets #7, Count: 0
Buckets #8, Count: 0
Buckets #9, Count: 0
Buckets #10, Count: 0
Buckets #11, Count: 0
Buckets #12, Count: 0
Buckets #13, Count: 0
Buckets #14, Count: 0
Buckets #15, Count: 0

ResourceMetrics #0
Resource SchemaURL: 
Resource attributes:
     -> telemetry.sdk.language: Str(python)
     -> telemetry.sdk.name: Str(opentelemetry)
     -> telemetry.sdk.version: Str(1.34.1)
     -> service.name: Str(unknown_service)
ScopeMetrics #0
ScopeMetrics SchemaURL: 
InstrumentationScope example-meter 
Metric #0
Descriptor:
     -> Name: http_requests_total
     -> Description: Total number of HTTP requests
     -> Unit: 1
     -> DataType: Sum
     -> IsMonotonic: true
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> service.name: Str(demo-service)
     -> env: Str(dev)
StartTimestamp: 2025-06-21 08:10:25.2128074 +0000 UTC
Timestamp: 2025-06-21 08:10:45.2229319 +0000 UTC
Value: 10
Metric #1
Descriptor:
     -> Name: http_response_duration_seconds
     -> Description: Response duration in seconds
     -> Unit: s
     -> DataType: Histogram
     -> AggregationTemporality: Cumulative
HistogramDataPoints #0
Data point attributes:
     -> service.name: Str(demo-service)
     -> env: Str(dev)
StartTimestamp: 2025-06-21 08:10:25.2128411 +0000 UTC
Timestamp: 2025-06-21 08:10:45.2229319 +0000 UTC
Count: 10
Sum: 3.250000
Min: 0.100000
Max: 0.550000
ExplicitBounds #0: 0.000000
ExplicitBounds #1: 5.000000
ExplicitBounds #2: 10.000000
ExplicitBounds #3: 25.000000
ExplicitBounds #4: 50.000000
ExplicitBounds #5: 75.000000
ExplicitBounds #6: 100.000000
ExplicitBounds #7: 250.000000
ExplicitBounds #8: 500.000000
ExplicitBounds #9: 750.000000
ExplicitBounds #10: 1000.000000
ExplicitBounds #11: 2500.000000
ExplicitBounds #12: 5000.000000
ExplicitBounds #13: 7500.000000
ExplicitBounds #14: 10000.000000
Buckets #0, Count: 0
Buckets #1, Count: 10
Buckets #2, Count: 0
Buckets #3, Count: 0
Buckets #4, Count: 0
Buckets #5, Count: 0
Buckets #6, Count: 0
Buckets #7, Count: 0
Buckets #8, Count: 0
Buckets #9, Count: 0
Buckets #10, Count: 0
Buckets #11, Count: 0
Buckets #12, Count: 0
Buckets #13, Count: 0
Buckets #14, Count: 0
Buckets #15, Count: 0

ResourceMetrics #0
Resource SchemaURL: 
Resource attributes:
     -> telemetry.sdk.language: Str(python)
     -> telemetry.sdk.name: Str(opentelemetry)
     -> telemetry.sdk.version: Str(1.34.1)
     -> service.name: Str(unknown_service)
ScopeMetrics #0
ScopeMetrics SchemaURL: 
InstrumentationScope example-meter 
Metric #0
Descriptor:
     -> Name: http_requests_total
     -> Description: Total number of HTTP requests
     -> Unit: 1
     -> DataType: Sum
     -> IsMonotonic: true
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> service.name: Str(demo-service)
     -> env: Str(dev)
StartTimestamp: 2025-06-21 08:11:08.712552 +0000 UTC
Timestamp: 2025-06-21 08:11:13.7145489 +0000 UTC
Value: 3
Metric #1
Descriptor:
     -> Name: http_response_duration_seconds
     -> Description: Response duration in seconds
     -> Unit: s
     -> DataType: Histogram
     -> AggregationTemporality: Cumulative
HistogramDataPoints #0
Data point attributes:
     -> service.name: Str(demo-service)
     -> env: Str(dev)
StartTimestamp: 2025-06-21 08:11:08.7125832 +0000 UTC
Timestamp: 2025-06-21 08:11:13.7145489 +0000 UTC
Count: 3
Sum: 0.450000
Min: 0.100000
Max: 0.200000
ExplicitBounds #0: 0.000000
ExplicitBounds #1: 5.000000
ExplicitBounds #2: 10.000000
ExplicitBounds #3: 25.000000
ExplicitBounds #4: 50.000000
ExplicitBounds #5: 75.000000
ExplicitBounds #6: 100.000000
ExplicitBounds #7: 250.000000
ExplicitBounds #8: 500.000000
ExplicitBounds #9: 750.000000
ExplicitBounds #10: 1000.000000
ExplicitBounds #11: 2500.000000
ExplicitBounds #12: 5000.000000
ExplicitBounds #13: 7500.000000
ExplicitBounds #14: 10000.000000
Buckets #0, Count: 0
Buckets #1, Count: 3
Buckets #2, Count: 0
Buckets #3, Count: 0
Buckets #4, Count: 0
Buckets #5, Count: 0
Buckets #6, Count: 0
Buckets #7, Count: 0
Buckets #8, Count: 0
Buckets #9, Count: 0
Buckets #10, Count: 0
Buckets #11, Count: 0
Buckets #12, Count: 0
Buckets #13, Count: 0
Buckets #14, Count: 0
Buckets #15, Count: 0

ResourceMetrics #0
Resource SchemaURL: 
Resource attributes:
     -> telemetry.sdk.language: Str(python)
     -> telemetry.sdk.name: Str(opentelemetry)
     -> telemetry.sdk.version: Str(1.34.1)
     -> service.name: Str(unknown_service)
ScopeMetrics #0
ScopeMetrics SchemaURL: 
InstrumentationScope example-meter 
Metric #0
Descriptor:
     -> Name: http_requests_total
     -> Description: Total number of HTTP requests
     -> Unit: 1
     -> DataType: Sum
     -> IsMonotonic: true
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> service.name: Str(demo-service)
     -> env: Str(dev)
StartTimestamp: 2025-06-21 08:11:08.712552 +0000 UTC
Timestamp: 2025-06-21 08:11:18.7345739 +0000 UTC
Value: 6
Metric #1
Descriptor:
     -> Name: http_response_duration_seconds
     -> Description: Response duration in seconds
     -> Unit: s
     -> DataType: Histogram
     -> AggregationTemporality: Cumulative
HistogramDataPoints #0
Data point attributes:
     -> service.name: Str(demo-service)
     -> env: Str(dev)
StartTimestamp: 2025-06-21 08:11:08.7125832 +0000 UTC
Timestamp: 2025-06-21 08:11:18.7345739 +0000 UTC
Count: 6
Sum: 1.350000
Min: 0.100000
Max: 0.350000
ExplicitBounds #0: 0.000000
ExplicitBounds #1: 5.000000
ExplicitBounds #2: 10.000000
ExplicitBounds #3: 25.000000
ExplicitBounds #4: 50.000000
ExplicitBounds #5: 75.000000
ExplicitBounds #6: 100.000000
ExplicitBounds #7: 250.000000
ExplicitBounds #8: 500.000000
ExplicitBounds #9: 750.000000
ExplicitBounds #10: 1000.000000
ExplicitBounds #11: 2500.000000
ExplicitBounds #12: 5000.000000
ExplicitBounds #13: 7500.000000
ExplicitBounds #14: 10000.000000
Buckets #0, Count: 0
Buckets #1, Count: 6
Buckets #2, Count: 0
Buckets #3, Count: 0
Buckets #4, Count: 0
Buckets #5, Count: 0
Buckets #6, Count: 0
Buckets #7, Count: 0
Buckets #8, Count: 0
Buckets #9, Count: 0
Buckets #10, Count: 0
Buckets #11, Count: 0
Buckets #12, Count: 0
Buckets #13, Count: 0
Buckets #14, Count: 0
Buckets #15, Count: 0

ResourceMetrics #0
Resource SchemaURL: 
Resource attributes:
     -> telemetry.sdk.language: Str(python)
     -> telemetry.sdk.name: Str(opentelemetry)
     -> telemetry.sdk.version: Str(1.34.1)
     -> service.name: Str(unknown_service)
ScopeMetrics #0
ScopeMetrics SchemaURL: 
InstrumentationScope example-meter 
Metric #0
Descriptor:
     -> Name: http_requests_total
     -> Description: Total number of HTTP requests
     -> Unit: 1
     -> DataType: Sum
     -> IsMonotonic: true
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> service.name: Str(demo-service)
     -> env: Str(dev)
StartTimestamp: 2025-06-21 08:11:08.712552 +0000 UTC
Timestamp: 2025-06-21 08:11:23.752477 +0000 UTC
Value: 8
Metric #1
Descriptor:
     -> Name: http_response_duration_seconds
     -> Description: Response duration in seconds
     -> Unit: s
     -> DataType: Histogram
     -> AggregationTemporality: Cumulative
HistogramDataPoints #0
Data point attributes:
     -> service.name: Str(demo-service)
     -> env: Str(dev)
StartTimestamp: 2025-06-21 08:11:08.7125832 +0000 UTC
Timestamp: 2025-06-21 08:11:23.752477 +0000 UTC
Count: 8
Sum: 2.200000
Min: 0.100000
Max: 0.450000
ExplicitBounds #0: 0.000000
ExplicitBounds #1: 5.000000
ExplicitBounds #2: 10.000000
ExplicitBounds #3: 25.000000
ExplicitBounds #4: 50.000000
ExplicitBounds #5: 75.000000
ExplicitBounds #6: 100.000000
ExplicitBounds #7: 250.000000
ExplicitBounds #8: 500.000000
ExplicitBounds #9: 750.000000
ExplicitBounds #10: 1000.000000
ExplicitBounds #11: 2500.000000
ExplicitBounds #12: 5000.000000
ExplicitBounds #13: 7500.000000
ExplicitBounds #14: 10000.000000
Buckets #0, Count: 0
Buckets #1, Count: 8
Buckets #2, Count: 0
Buckets #3, Count: 0
Buckets #4, Count: 0
Buckets #5, Count: 0
Buckets #6, Count: 0
Buckets #7, Count: 0
Buckets #8, Count: 0
Buckets #9, Count: 0
Buckets #10, Count: 0
Buckets #11, Count: 0
Buckets #12, Count: 0
Buckets #13, Count: 0
Buckets #14, Count: 0
Buckets #15, Count: 0

ResourceMetrics #0
Resource SchemaURL: 
Resource attributes:
     -> telemetry.sdk.language: Str(python)
     -> telemetry.sdk.name: Str(opentelemetry)
     -> telemetry.sdk.version: Str(1.34.1)
     -> service.name: Str(unknown_service)
ScopeMetrics #0
ScopeMetrics SchemaURL: 
InstrumentationScope example-meter 
Metric #0
Descriptor:
     -> Name: http_requests_total
     -> Description: Total number of HTTP requests
     -> Unit: 1
     -> DataType: Sum
     -> IsMonotonic: true
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> service.name: Str(demo-service)
     -> env: Str(dev)
StartTimestamp: 2025-06-21 08:11:08.712552 +0000 UTC
Timestamp: 2025-06-21 08:11:28.7219619 +0000 UTC
Value: 10
Metric #1
Descriptor:
     -> Name: http_response_duration_seconds
     -> Description: Response duration in seconds
     -> Unit: s
     -> DataType: Histogram
     -> AggregationTemporality: Cumulative
HistogramDataPoints #0
Data point attributes:
     -> service.name: Str(demo-service)
     -> env: Str(dev)
StartTimestamp: 2025-06-21 08:11:08.7125832 +0000 UTC
Timestamp: 2025-06-21 08:11:28.7219619 +0000 UTC
Count: 10
Sum: 3.250000
Min: 0.100000
Max: 0.550000
ExplicitBounds #0: 0.000000
ExplicitBounds #1: 5.000000
ExplicitBounds #2: 10.000000
ExplicitBounds #3: 25.000000
ExplicitBounds #4: 50.000000
ExplicitBounds #5: 75.000000
ExplicitBounds #6: 100.000000
ExplicitBounds #7: 250.000000
ExplicitBounds #8: 500.000000
ExplicitBounds #9: 750.000000
ExplicitBounds #10: 1000.000000
ExplicitBounds #11: 2500.000000
ExplicitBounds #12: 5000.000000
ExplicitBounds #13: 7500.000000
ExplicitBounds #14: 10000.000000
Buckets #0, Count: 0
Buckets #1, Count: 10
Buckets #2, Count: 0
Buckets #3, Count: 0
Buckets #4, Count: 0
Buckets #5, Count: 0
Buckets #6, Count: 0
Buckets #7, Count: 0
Buckets #8, Count: 0
Buckets #9, Count: 0
Buckets #10, Count: 0
Buckets #11, Count: 0
Buckets #12, Count: 0
Buckets #13, Count: 0
Buckets #14, Count: 0
Buckets #15, Count: 0

devsekar-1992 avatar Jun 21 '25 08:06 devsekar-1992

Thanks @devsekar-1992 I now understand a little better. I'm not an expert on metrics, but it does look like your hypothesis is correct: prometheus doesn't support delta temporality, and that's our default (at least, for the Counter instrument).

The easy fix for you to make it work with prometheus should be to provide a temporality preference to the exporter:

$reader = new ExportingReader(new MetricExporter($transport, \OpenTelemetry\SDK\Metrics\Data\Temporality::CUMULATIVE));

I did find this in the spec:

OTLP Metrics Exporter MUST provide configuration to influence the MetricReader output temporality as a function of instrument kind. This option MAY be named temporality, and MUST set temporality preference to Cumulative for all instrument kinds by default.

That last part sounds like all instruments should be Cumulative by default, however it's specifically talking about the otlp exporter. I'm not sure if the spec has changed since this was implemented, but what we do is defer to the instruments themselves if a preference wasn't set on the exporter, and synchronous instruments seem to default to Delta (I can't find any part of the spec which defines this) - @Nevay ?

edit: I think our default implementation might be based on LowMemory temporality

brettmc avatar Jun 24 '25 03:06 brettmc

I think our default implementation might be based on LowMemory temporality

Yes, we use LowMemory by default (declarative and env based config are already using Cumulative by default).

it seems that Alloy is based on the opentelemetry collector

An alternative fix would be to use the delta to cumulative processor.

not sure if the spec has changed since this was implemented

Our metrics implementation was written before LowMemory was added to the spec / the default temporality was changed from SHOULD to MUST ~11 months ago. Note that our interpretation of Delta and LowMemory is also out of sync with the spec (UpDownCounters should always use Cumulative temporality). Not sure whether this should/can be fixed in 1.x, but we should definitely fix it for 2.0.

Nevay avatar Jun 24 '25 20:06 Nevay

@devsekar-1992 I've created an issue to update our metrics implementation to current spec requirements, but there's no ETA on when that might get fixed. In the meantime, did setting temporality preference on the exporter help? Is there any example code or documentation you think could be updated to alert others about delta vs cumulative temporality when using Prometheus?

brettmc avatar Jun 27 '25 06:06 brettmc

Alloy’s otelcol components are mostly wrappers for the OpenTelemetry Collector, but the Prometheus exporter is an exception:

https://github.com/grafana/alloy/blob/dcd92d4692c075bc18073dccaf0cd8762c2018ca/docs/sources/reference/components/otelcol/otelcol.exporter.prometheus.md?plain=1#L18

There are some opinionated choices documented in the code comments:

https://github.com/grafana/alloy/blob/dcd92d4692c075bc18073dccaf0cd8762c2018ca/internal/component/otelcol/exporter/prometheus/internal/convert/convert.go#L431-L433

https://github.com/grafana/alloy/blob/dcd92d4692c075bc18073dccaf0cd8762c2018ca/internal/component/otelcol/exporter/prometheus/internal/convert/convert.go#L477-L479

I’d suggest reading the other code comments there, as they were helpful to me in figuring out how to configure the OTel PHP exporter to get metrics through Alloy into Prometheus. That said, specifying cumulative temporality as @brettmc suggested has been working for me.

smaddock avatar Jun 27 '25 09:06 smaddock

I think I've got this working per spec in #1715 - the default for any instrument should be cumulative, which is what the spec now says. This will be in SDK v2, which is getting closer to being released :)

brettmc avatar Sep 11 '25 05:09 brettmc

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale[bot] avatar Oct 18 '25 05:10 stale[bot]