prometheus
Decision needed: Should we allow deleting a labelset from an existing metric?
This issue comes from this recently closed PR.
I'll try to summarize the situation here, but I encourage to read the discussion on that PR.
Basically, the question is whether we should allow deletion of a labelset for a metric that's observed values on that labelset.
Reasons for doing it:
- Removing stale metrics
- Example use case: Exporting metrics about containers running on a host, with container identifier as a guid that is part of the label set. Containers disappear after being terminated by our cluster manager and we'd like the related metrics to disappear as well.
- The best practices actually say this should be a feature:
Metrics with labels SHOULD support a remove() method with the same signature as labels() that will remove a Child from the metric no longer exporting it
Reasons against:
- This should be done using const metrics, as supported by the Go/Python/Java do. See https://www.robustperception.io/setting-a-prometheus-counter/. Maybe we need to add this instead to the client
- That bit of the best practices quoted above likely needs some reconsidering. There are use cases for this, but they're very very rare. Mostly the people looking for this actually want const metrics.
- This is actually quite involved to implement with our new file-based Data Store. This is not really a reason not to do this, but it's a cost that is good to keep in mind. The new implementation is considerably harder than the code change in the PR linked above.
An example found in the wild of someone needing this feature. I'm not sure what's the best answer, just adding context: https://stackoverflow.com/questions/58688050/handling-stale-data-in-prometheus-gauges
Note: Java has a Gauge.clear() method that kills all labelsets for a gauge.
That question is looking for custom collectors as they're writing an exporter, direct instrumentation is the wrong way to handle it so isn't relevant here.
