David Ashpole
David Ashpole
cc @open-telemetry/wg-prometheus
> I'm concerned that we're offering an easy way to break the OTel data model instead of fixing a memory leak. I don't think we've seen any evidence of a...
I have a soak test running now. It generates 100 new series each scrape (and stops emitting the old series), which should eventually make the collector OOM if there is...
My test seems to confirm that as well. Memory has stayed constant for the past 12 hours.
They are all counter series. If there was a leak in the garbage collection, those should demonstrate it.
Label length shouldn't matter too much. We just care about start time tracking, which is done for all cumulative types (counter, histogram, summary)
And each scrape, it produces different labels than the previous scrape.
It is mostly a code design problem. Staleness is tracked in the prometheus library, and we track the start time separately in the prometheus receiver. It is probably also an...
Looking at the prom implementation of staleness, it looks like a label set has a [Hash() function](https://github.com/prometheus/prometheus/blob/b8af4632be96081f60d280971d7ddc298d27a006/scrape/scrape.go#L1531) that it uses to track each series. We keep the entire OTLP representation...
I've opened https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/13451 to track the fingerprinting improvement discussed above.