OpenMetrics Exemplars

[bilbof]

Title: Add support for OpenMetrics exemplars

Description:

Currently Envoy supports OpenMetrics format but does not export exemplars. Exemplars are very useful for discovering traces.

Envoy exports traces in a few different formats. There's also currently an effort to support OpenTelemetry for tracing https://github.com/envoyproxy/envoy/issues/9958.

https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#exemplars-1

[kyessenov]

Seems reasonable. Are you referring to /stats/prometheus endpoint for OpenMetrics integration?

[bilbof]

Yes exactly

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

[bilbof]

No stalebot

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

[github-actions[bot]]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions.

[wesbrownfavor]

Did you ever find a workaround for this?

[moderation]

I was originally go to say that the Prometheus proto at https://github.com/envoyproxy/envoy/blob/main/api/bazel/repository_locations.bzl#L89-L101 needs to be updated to the latest commit. But that repo is deprecated. I think it got histograms but I don't think examplars ever landed.

I think the real answer is to upgrade to https://github.com/OpenObservability/OpenMetrics. This might be a fair bit of work but makes sense for the project IMO

[therealak12]

I want to implement this feature. I appreciate any help and guidance.

At least the following modifications should be made. Please correct me if I'm wrong. Envoy doesn't use any Prometheus client libraries. Instead, it implements its own Counter, Gauge, Histogram, and TextReadout. These types can be rendered in different formats, one of them is the Prometheus exposition format. So:

• Add proper fields and getter/setter methods to the MetricHelper and MetricImpl classes for accommodating exemplars.
• A new completeWithExemplar method should be added to CompletableTiemspan to store the timestamp and exemplar-related labels.
• Modify the generateHistogramOutput function to include timestamp and exemplar-related labels for histograms.
• Modify the generateStatNumericOutput function to include timestamp and exemplar-related labels for counters.

[therealak12]

Today, I began working on implementing the new feature. I discovered that Envoy uses libcircllhist for histograms.

To add support for exemplars, changes need to be made to circllhist before they can be integrated into Envoy. Additionally, the modifications required in Envoy's code are more extensive than I initially anticipated. I will continue to work on this, but I'm not sure if I have enough time to complete the implementation.