opentelemetry-go
opentelemetry-go copied to clipboard
open-telemetry
OpenTelemetry does not support resetting in-memory Metric to clean the memory usage
Problem Statement
We are using Kyverno as a Policy engine in the Kubernetes environment in our product. We often see an increase in the memory consumption of the Kyverno pod due to collected in-memory metrics which have high cardinality due to many labels.
Up-to and including release-1.7 of Kyverno it was using the Prometheus library directly to expose Metrics. Prometheus client library supports reseting metric vectors which were used by Kyverno to periodically (configurable period) reset its internal metrics registry to clean its memory usage and footprint associated with metrics.
As of release-1.8 Kyverno started to support OpenTelemetry. After this transition this feature of reseting internal metrics to clean its memory usage is lost as OpenTelemetry does not allow reseting metrics, which is supported by the underlying Prometheus library.
Proposed Solution
OpenTelemetry should support reseting the in-memory metric vectors allowing to clean its memory usage. As this is supported by Prometheus to reset the Metric vectors and OpenTelemetry is using Prometheus client library hence, it would be very beneficial for us to consume this feature again in the Kyverno project.
Alternatives
No alternative solution
Prior Art
A clear and concise list of any similar and existing solutions from other projects that provide context to possible solutions.
Additional Context
As mentioned the underlying Prometheus library supports reseting in-memory Metric vectors to clean the memory usage.
This feature has to be first specified in OpenTelemetry Specification. I suggest creating an issue in https://github.com/open-telemetry/opentelemetry-specification first.
@pellared thanks for your input. I created a new issue in the opentelemetry-specification repository now.
Related issue:
- https://github.com/open-telemetry/opentelemetry-specification/issues/2232