OpenMetrics `_created` timestamp

[shakuzen]

See the OpenMetrics specification. This is a follow-up to the initial tracking issue #2419.

We don't currently track or publish the created timestamp for each MetricFamily.

[alshain]

This seems to landing soon in Prometheus, so it'd be great to see it supported in Micrometer.

https://github.com/prometheus/prometheus/pull/13506

[shakuzen]

Thanks for the heads up. I've subscribed to updates on that PR. I'm not sure if we need to do anything in Micrometer for this anymore since we've upgrade to the Prometheus Java client 1.x (see #4406). The new client has an exporter configuration for created timestamps - see https://prometheus.github.io/client_java/config/config/#exporter-properties. I suspect that would work with Micrometer 1.13.0 (currently 1.13.0-RC1 is available from the milestone repository). Does that do what you need?

[stefan-fast]

I looked into this and tried using it. Indeed, it seems to be implemented in the lastest versions of the Prometheus Java Client and Micrometer is using these new versions with support for includeCreatedTimestamps. Unfortunately, the createdTimestampMillis parameter is always set to 0L in micrometer for supported metrics types, e.g. counters: https://github.com/micrometer-metrics/micrometer/blob/be70882669c690b28f8ec1c6143edc2fb33a7cb2/implementations/micrometer-registry-prometheus/src/main/java/io/micrometer/prometheusmetrics/PrometheusMeterRegistry.java#L221 (last parameter in that line). Therefore, the hasCreatedTimestamp() check in the Prometheus Java Client is always false: https://github.com/prometheus/client_java/blob/e17c0a10d5b55afc107b2e0114a433742b96a0ab/prometheus-metrics-exposition-formats/src/main/java/io/prometheus/metrics/expositionformats/PrometheusTextFormatWriter.java#L102 Because of that, the _created timestamp is never actually written yet. In summary, if I didn't miss anything, micrometer still needs to implment _created timestamp support.

Are there any plans to support it in the near future?

[jonatan-ivanov]

@stefan-fast Yes, that's what this issue is about: we don't track the created timestamp right now. We are going to do some planning for the next feature release, I will drop this into the set of topics we will discuss but it seems right now there is not a lot of users who needs this

Is there a certain use-case you need _created for or is the lack of it causes issues for you or is it a nice to have since the specs says it should be there?

[stefan-fast]

@jonatan-ivanov Thanks for taking this into the planning discussions!

Yes, basically this blog post by Arthur Sens sums up and explains the problem pretty well: https://www.arthursens.dev/posts/created-timestamps

I have counters that increase fast before the first scrape by prometheus. So the first increase from 0 to this first scrape value X is missing. Using increase or rate functions on this metric will then result in wrong values to be displayed. E.g. I did an increase over a longer timespan to get the total count of requests in that timeframe. Because the counter only increased before the first scrape and not after anymore, the increase function over the timespan resulted in 0 instead of the real correct value.

[alshain]

We ran into the same thing. Also, we have metrics with labels that depend on dynamic labels, such as the exception class.

So even if the application has been scraped for hours, these new metrics will appear with an initial count of 1, and due to the lack of a previous scrape and no _created timestamp, alerts won't trigger.

[Hatelix]

Me and my team at work are eagerly awaiting the support for created timestamps.

[sbwdb]

We ran into the same issue. We would highly appreciate support for the created timestamps.

[shakuzen]

I've opened #6032 for this. Review would be appreciated.

[shakuzen]

This is available in 1.15.0-SNAPSHOT now. If any of the users that were asking for it would try it out and give feedback, that would be helpful. The created timestamps can be enabled with the Prometheus Java client property io.prometheus.exporter.includeCreatedTimestamps , which can be configured as mentioned in this section of the documentation. Or if you're using Spring Boot, it can be configured via configuration properties using e.g. management.prometheus.metrics.export.properties.io.prometheus.exporter.includeCreatedTimestamps=true in application.properties.