prometheus
Native Summaries
Proposal
This requires a bigger proposal, MVP and motivation, but I wanted to officially start early discussions and potential work here.
The idea is to introduce the native summaries -- the replacement for "classic summaries" we have now that is build from multiple counter series. Native summaries would be contained as a single series that's more efficient and transactional.
PromQL could be adapted in similar fashion to native histogram PromQL syntax for consistency (also reusing the https://prometheus.io/docs/prometheus/latest/querying/functions/#histogram_count-and-histogram_sum functions). However due to lower, mostly historic use of summaries in the ecosystem (see Considerations), perhaps it would be easier and sufficient to only emulate classic view while storing native summaries under the hood (similar to https://github.com/prometheus/prometheus/issues/16948 idea).
Motivation
Native histograms were created for many reasons, but one of them was storage efficiency and transactionality. With native histogram representation, you don't have potentially 30 series for one histogram, but one which makes for incredible benefits around indexing and storage, despite more beefier sample size (float vs struct).
As per transactionality, having one series and not 30 also gives guarantee that all parts of the Prometheus (and ecosystem) will see each part of histogram (buckets, sum, count) exactly at once. This is especially important for distributed systems with remote write and sharding, as well as querying vs scraping drift etc (when no isolation is possible e.g. on Thanos).
The same efficiency and transactionality problem exists for classic summaries as well, solved by adding the native summaries.
Considerations
- Summaries are still used widely, although less than other types. However we tend to NOT recommend them in practice, especially with the new improved native histograms. However they do still exists and are not deprecated (and there is not plan for that as of now).
- Native histograms brought sparseness and exponential bucketing; there is no such dimension in native summaries planned -- which makes them easier to implement.
WDYT @krajorama @beorn7 @bboreham @roidelapluie @RichiH
Just for clarity on the scope, you're suggesting native summaries with explicitly set quantiles (like native histograms custom buckets) and no automatic quantiles (like native histograms exponential buckets), right? Implying no change to SDK, only to promQL (maybe) from user point of view.
I'm not against the idea, but note that the current code around native histograms means we have a lot of switch statements and triplicated (in one case 5 times copied) code and a lot of special cases for performance reasons. Ideally before we introduce a new type (subtype?) we think about how to make the code more maintainable, I really need to add an issue for this. Also there's #16581, #15177.
Along the lines of what @krajorama said about code structure, see also parts of https://github.com/prometheus/prometheus/issues/12233#issuecomment-2025848266 . #12233 is also a proposal to add a new native metric type (but one that doesn't exist in a non-native form, so arguably it is a larger effort than native summaries, but the general considerations about how to make adding more native types feasible apply).
I have been thinking about "native summaries" for a while, so here I'll try to dump a number of random thoughts lingering in my brain, sorted from more concrete and tangible to distant visions for the future of PromQL. (This is not meant to distract from the the implementation aspect mentioned above.)
Do we need this at all?
The need for native summaries is way less pressing than it was for native histograms. Native histograms enabled usage patterns that were just infeasible before. Native summaries would mostly make things more consistent and get rid of a few more magic labels and suffixes, avoiding name collisions and the dreadful floats as string labels – which would give me a lot of peace of mind, but it leaves the question if it is worth the considerable effort.
Can we just add pre-calculated quantiles as "fields" to the existing native histograms?
Since histogram and summaries share the count and sum "fields" (represented as separate series in the non-native types), the thought of combining them both into a combined "distribution" type has been haunting me from the beginning of written history. This type would have count and sum and then optionally any number of buckets buckets of a (currently one) certain schema (not sure if we would ever consider multi-schema histograms…) and optionally any number of pre-calculated quantiles (maybe this time including metadata from the beginning for which timeframe they are calculated, not just the phi value itself). It would be possible to have both. I could even see use cases for the combined case, but even without that, it might simplify both usage and implementation.
IIRC OpenMetrics kept summaries and histograms deliberately distinct, even regulating that there can be quantile-less summaries, but not bucket-less histograms (but the latter might just be because of the redundant +Inf bucket). So maybe there are good reasons for keeping them separate that I don't see right now.
Should we establish a stricter type system first?
This is probably a Prom v4 thing, but maybe that's not that far away.
With the "dynamically typed" approach we took for NH, series can have mixed type (switching from float samples to NH samples, and the latter even switching from counter histogram to gauge histogram). This was mostly motivated by the current semantics where a series does not have a fixed type (so far just counter vs. gauge vs. untyped/unknown), but it got much juicier because previously the data we stored was always just a float, while now we have to change from float to counter histogram to gauge histogram, and even within those between integer histograms and float histograms (where the latter is more an optimization of storage and exposition format than a type from the perspective of PromQL semantics). This created a lot of complications in the code because we have to deal with those mixed-type series in storage and querying. It even affects the user because PromQL functions need to have defined behavior for acting on the different types, and what happens if they encounter a mix of types.
The "statically typed" approach would be to make the type part of the series's identity, thereby avoiding mixed type series. (However, this would not apply to mix of integer and float histograms, which would still require shenanigans in the code.) We could even consider propagating the type to PromQL syntax, so that e.g. applying a function that only acts on float counters to anything else (gauge counter, a histogram of any flavor) would be a syntax error.
Many things have to be fleshed out before putting this into practice, but depending on how we implement it exactly, PromQL and/or code could become clearer and simpler by avoiding mixed series. This might be relevant for the code structure problems raised above. And to circle back: That's the reason why I'm mentioning this point here, maybe doing this change first will significantly simplify the work needed to implement native summaries.
Should we maybe revamp metric types more fundamentally?
This is now completely visionary, probably Prom v5 scope, but it connects some dots from above, so let me write it down here to get it on file.
The fundamental idea here is to make every sample structured and, if you want, "polymorphic" (at least logically – how things are stored in optimized fashion under the hood is another question). The different "fields" could be used to store all kind of things, and it would be easy to extend the data model here. A current example is that every sample could have a field "created-at timestamp" (based on something @bwplotka told me as desirable and how it works in Monarch). But the fields could also contain the precalculated quantiles, or histogram buckets of various kinds, or the count or the sum of a distribution, or a plain float counter value, or a plain float gauge value. Some of these fields needed complex parametrized accessor methods (like buckets, which can be highly dynamic, or quantiles, which require at least the quantile value, but maybe also the covered timeframe, and maybe you don't want to ask "give the the 99th percentile over the last 10m" but more like "what precalculated quantiles do you have?"). But other need "simple" accessors. And then we get a syntax that looks familiar. If foo is such a structured "polymorphic" value, you could access its gauge value (if there is one) with foo.gauge, its counter value (if there is one) with foo.total, its distribution count and sum with foo.count and foo.sum, and one or more buckets maybe with foo.bucket(...some_parameters...) (returning something complex on its own). PromQL functions, in turn, could use these explicit accessors, like rate(foo.total[10m]), and then rate(foo.gauge[10m]) could return a syntax error as considered above. And functions could pick what they need, so rate(foo[10m]) would dynamically look for "rate'able" fields like total, count, sum, buckets, and return a result with those fields rate'd, so they are now gauges (with .gauge for float gauges, but we will need the corresponding histogram fields for a gauge histogram, maybe gbuckets/gsum,/gcount). In that way, you could have a sample that is at the same time a float counter, a float gauge, a histogram and a summary. (Not that that would happen often in practice for all of them, but for some fields, I can totally see that, cf. https://github.com/prometheus/client_golang/issues/829 .) And, again circling back, we would implicitly get "native summaries" in the form of .quantile(...) fields.
Hi folks, I’m new to the Prometheus repo and exploring the Native Histogram flow to understand how Native Summaries could fit in. I think it’s important to first fix the code structure issues (like switch duplications, as mentioned) to make adding new native types smoother.
Just sharing my thoughts as I get familiar. Could you guide me if I can pick up:
- Writing a skeleton struct for Native Summary samples (model/summary.go)?
- Adding edge-case tests for Native Histograms (so I can understand the flow)?
- Helping with refactoring code duplication (switch-case logic) that was mentioned?
Any guidance would be appreciated on where I can begin contributing meaningfully.
Thanks!
Thanks for your interest, @DevAnuragT . I would say the refactoring of the code we need here is a very challenging task, even for the most experienced Prometheus developers. I would definitely not recommend any newcomer to get close to it.
Once we have a design doc for native summaries, there might be very will tasks that are good for beginners or intermediate experience levels. But we aren't there yet.
In the meantime, I recommend to look for issues tagged with "good first issue", or even better find an issue that you already have good domain knowledge for. (For example, any decent React coder could pick up a bunch of UI issues.)
Should we maybe revamp metric types more fundamentally?
This is now completely visionary, probably Prom v5 scope, but it connects some dots from above, so let me write it down here to get it on file.
The fundamental idea here is to make every sample structured and, if you want, "polymorphic" (at least logically – how things are stored in optimized fashion under the hood is another question). The different "fields" could be used to store all kind of things, and it would be easy to extend the data model here.
That could even offer an opportunity to enable emitters to declare which labels are identifying vs informational on series, particularly info-style metrics. This might open a path for Prometheus to improve how it stores labels that are not expected to be used to search for and match series. There's some discussion in https://github.com/prometheus/prometheus/issues/9139 about the possibility of storing "non-cardinal" labels a different way, reducing the impact of high-churn and long label values being (ab)used to carry string payloads on the TSDB. One of the problems there is that Prometheus is schemaless and largely type-less, so it's difficult to handle labels that might be stored separately at query-time.
Improving handling of enumerations (StateSets), a long standing area of pain for Prometheus users (IMHO) might be another opportunity created by a more flexible type system, by supporting a native OpenMetrics StateSet representation. See also this discussion in the OpenTelemetry spec repo that touches on how Prometheus stores enumerations: https://github.com/open-telemetry/opentelemetry-specification/issues/1711
Hi @beorn7 , I’m interested in Prometheus and participating in LFX as part of it in the upcoming term. I was curious if you have any suggestions on how I should get started, and how the selection process works?
