feat: Support Prometheus Created Timestamps

[ArthurSens]

Description: exporter/prometheusexporter now supports Prometheus Created Timestamps.

Link to tracking Issue: Fix #32521

Testing: Unit tests were extended with the new scope

Documentation: No documentation was added so far, looking for direction about where would be the best place for it :)

[ArthurSens]

I'm adding a replace to go mod because we require a new version of https://github.com/prometheus/client_golang. The new version should be out next week :)

[ArthurSens]

prometheus/client_golang 1.20 was released a few hours ago, PR ready for review!

[ArthurSens]

Urgh, appologies for the big review-request 😬... I'm not sure how to proceed here.

The change requires a bump in prometheus/client_golang, but I'm struggling to make this bump and having the tests to pass at the same time

[dashpole]

hmmm, you can try bumping the prometheus dependency in a separate PR if you want. Otherwise, renovate should create a PR to do that in the next few days (similar to https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/33032)

[dashpole]

it does look like this added client_golang as a dependency to a lot of modules that didn't have it before. Maybe it just needs another tidy?

[andrzej-stencel]

Thank you @ArthurSens for this contribution.

Please help me understand, what is the expected change in the behavior of the Prometheus exporter from user perspective, if any? Can you give an example collector configuration and describe the difference in the exporter's output with this change vs. without it? I'm not seeing any changelog entry or documentation update explaining this.

I tried the following configuration:

exporters:
  prometheus:
    endpoint: localhost:1234

receivers:
  hostmetrics:
    scrapers:
      cpu:

service:
  pipelines:
    metrics:
      exporters:
        - prometheus
      receivers:
        - hostmetrics

The produced metric system.cpu.time is a cumulative monotonic sum, but I didn't notice any change in the output of the Prometheus exporter.

$ curl localhost:1234/metrics
# HELP system_cpu_time_seconds_total Total seconds each logical CPU spent on each mode.
# TYPE system_cpu_time_seconds_total counter
system_cpu_time_seconds_total{cpu="cpu0",state="idle"} 23585.85
system_cpu_time_seconds_total{cpu="cpu0",state="interrupt"} 0
system_cpu_time_seconds_total{cpu="cpu0",state="nice"} 0.7
system_cpu_time_seconds_total{cpu="cpu0",state="softirq"} 15.05
system_cpu_time_seconds_total{cpu="cpu0",state="steal"} 0
system_cpu_time_seconds_total{cpu="cpu0",state="system"} 534.26
system_cpu_time_seconds_total{cpu="cpu0",state="user"} 1459.38
system_cpu_time_seconds_total{cpu="cpu0",state="wait"} 43
system_cpu_time_seconds_total{cpu="cpu1",state="idle"} 25029.56
system_cpu_time_seconds_total{cpu="cpu1",state="interrupt"} 0
system_cpu_time_seconds_total{cpu="cpu1",state="nice"} 0.45
system_cpu_time_seconds_total{cpu="cpu1",state="softirq"} 11.35
system_cpu_time_seconds_total{cpu="cpu1",state="steal"} 0
system_cpu_time_seconds_total{cpu="cpu1",state="system"} 221.28
system_cpu_time_seconds_total{cpu="cpu1",state="user"} 656.36
system_cpu_time_seconds_total{cpu="cpu1",state="wait"} 19.3
(...)

[ArthurSens]

Sure thing!

Created Timestamps is not considered stable in Prometheus yet, so we need to turn on feature flags depending on how you want to ingest it.

• Using OTel collector's prometheus receiver, you'd need to toogle the UseCreatedMetric feature gate[1].
• Using a Prometheus server, you'd need to turn on the created-timestamp-zero-ingestion feature flag[2].

Regarding the exposition format, created timestamps are only exposed when using the protobuf protocol. If you're using a Prometheus server with the feature-flag enabled, the protobuf format will automatically be set as the preferred protocol for all scrapes, as documented here[3].

There's work in progress to also expose it in the text format[4], but it will have to wait until the next release of client_golang 😬

[andrzej-stencel]

Sure thing!

Created Timestamps is not considered stable in Prometheus yet, so we need to turn on feature flags depending on how you want to ingest it.

* Using OTel collector's prometheus receiver, you'd need to toogle the `UseCreatedMetric` feature gate[[1](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/prometheusreceiver#:~:text=Feature%20gates%3A,feature%20gate%20option%3A)].

* Using a Prometheus server, you'd need to turn on the `created-timestamp-zero-ingestion` feature flag[[2](https://prometheus.io/docs/prometheus/latest/feature_flags/#created-timestamps-zero-injection)].

Regarding the exposition format, created timestamps are only exposed when using the protobuf protocol. If you're using a Prometheus server with the feature-flag enabled, the protobuf format will automatically be set as the preferred protocol for all scrapes, as documented here[3].

There's work in progress to also expose it in the text format[4], but it will have to wait until the next release of client_golang 😬

Looking at the code in Prometheus receiver to handle the create timestamps https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/17498/files, I would expect this PR in Prometheus exporter to expose e.g. system_cpu_time_created series for the system.cpu.time metric. Is this expectation incorrect?

[ArthurSens]

Looking at the code in Prometheus receiver to handle the create timestamps https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/17498/files, I would expect this PR in Prometheus exporter to expose e.g. system_cpu_time_created series for the system.cpu.time metric. Is this expectation incorrect?

Yes if you look at the protobuf format. Text format still doesn't expose _created lines. WIP here: https://github.com/prometheus/client_golang/pull/1408

[ArthurSens]

PR rebased after client_golang was updated elsewhere. Sorry everybody for the unnecessary request review 😬