prometheus
Add support for native histograms
It's still an experimental feature, but it would be great if prom2json wouldn't choke on it.
So it looks like as a result of prom2json using mostly upstream, it seems to support it, but only for when it calls the endpoint itself.
[
{
"name": "go_build_info",
"help": "Build information about the main Go module.",
"type": "GAUGE",
"metrics": [
{
"labels": {
"checksum": "",
"path": "github.com/hdost/example-prom",
"version": "(devel)"
},
"value": "1"
}
]
},
{
"name": "promhttp_metric_handler_errors_total",
"help": "Total number of internal errors encountered by the promhttp metric handler.",
"type": "COUNTER",
"metrics": [
{
"labels": {
"cause": "encoding"
},
"value": "0"
},
{
"labels": {
"cause": "gathering"
},
"value": "0"
}
]
},
{
"name": "rpc_durations_histogram_seconds",
"help": "RPC latency distributions.",
"type": "HISTOGRAM",
"metrics": [
{
"buckets": {
"-0.0001899999999999998": "4345",
"-0.0002899999999999998": "1766",
"-0.0003899999999999998": "630",
"-0.0004899999999999998": "170",
"-0.0005899999999999998": "39",
"-0.0006899999999999999": "11",
"-0.0007899999999999999": "3",
"-0.00089": "0",
"-0.00099": "0",
"-8.999999999999979e-05": "8396",
"0.00011000000000000022": "19037",
"0.00021000000000000023": "23322",
"0.0003100000000000002": "25882",
"0.0004100000000000002": "27114",
"0.0005100000000000003": "27526",
"0.0006100000000000003": "27650",
"0.0007100000000000003": "27688",
"0.0008100000000000004": "27690",
"0.0009100000000000004": "27691",
"1.0000000000000216e-05": "13694"
},
"count": "27691",
"sum": "0.33094722326916126"
}
]
},
{
"name": "rpc_durations_seconds",
"help": "RPC latency distributions.",
"type": "SUMMARY",
"metrics": [
{
"labels": {
"service": "exponential"
},
"quantiles": {
"0.5": "7.458277654072874e-07",
"0.9": "2.285574107047499e-06",
"0.99": "4.4695236807251784e-06"
},
"count": "41523",
"sum": "0.04146114661252858"
},
{
"labels": {
"service": "normal"
},
"quantiles": {
"0.5": "3.308699193165947e-05",
"0.9": "0.00027039975489900024",
"0.99": "0.00047890056212257883"
},
"count": "27691",
"sum": "0.33094722326916126"
},
{
"labels": {
"service": "uniform"
},
"quantiles": {
"0.5": "0.00010877566164376751",
"0.9": "0.0001793647812577891",
"0.99": "0.00019791887794769197"
},
"count": "20791",
"sum": "2.0847731502186093"
}
]
}
]
Example output.
The example output just shows classic buckets, not native buckets. The JSON rendering of prom2json is fairly custom. I'm afraid we need custom code to render the native histograms.
The example output just shows classic buckets, not native buckets. The JSON rendering of
prom2jsonis fairly custom. I'm afraid we need custom code to render the native histograms.
Yea I realized that after the fact, but I forgot to update 😵💫
Also based on your comment here, I am curious about one thing I know that buckets are bounded with open an closed. It seems like the boundaries are open towards zero and closed as their absolute value increases. Does this change with offset? I am wondering if it's worth showing or not. https://github.com/OpenObservability/OpenMetrics/issues/247#issue-1289064611
{
"count": 252719,
"sum": 2.588417777086236,
"buckets": [
"-0.0011613350732448448,-0.0010649489576809157": 1,
"-0.0008211879055212055,-0.0007530326295937211": 9,
"-0.0007530326295937211,-0.0006905339660024878": 50,
"-0.0006905339660024878,-0.0006332224387944383": 121,
"-0.0006332224387944383,-0.0005806675366224224": 213,
"-0.0005806675366224224,-0.0005324744788404579": 392,
"-0.0005324744788404579,-0.00048828125": 727,
"-0.00048828125,-0.0004477558804710308": 1171,
"-0.0004477558804710308,-0.00041059395276060273": 1672,
"-0.00041059395276060273,-0.00037651631479686053": 2193,
"-0.00037651631479686053,-0.0003452669830012439": 2903,
...
// Something like this?
"open:1.7080531329003556e-09,closed:1.862645149230957e-09": 1,
"open:2.6341780319308768e-09,closed:2.8725915130375712e-09": 2,
...
// Or maybe this
{
"lower": { "boundary": "open", "limit":"0.00048828125},
"upper": { "boundary": "closed", "limit":"0.0005324744788404579},
"count": 989
}
...
"0.00048828125,0.0005324744788404579": 989,
"0.0005324744788404579,0.0005806675366224224": 577,
"0.0005806675366224224,0.0006332224387944383": 298,
"0.0006332224387944383,0.0006905339660024878": 132,
"0.0006905339660024878,0.0007530326295937211": 53,
"0.0007530326295937211,0.0008211879055212055: 20,
"0.0008211879055212055,0.0008955117609420616: 7
]
}
That's a good question. And there is an even more fundamental one: Do we want to represent the buckets like in the exposition format (and how it is done internally), i.e. with spans and schema etc., so that bucket boundaries are implicit and not really readable for humans? Or do we want to convert them into individual buckets with explicit boundaries? We are doing the latter for the query API, see https://prometheus.io/docs/prometheus/latest/querying/api/#native-histograms . This does have a solution to your question about open and closed boundaries.
My current gut feeling is to make prom2json output as similar as possible to the output of the Prometheus query API. There might be some devil in the detail, but maybe you start with this goal and see how for you can get.
Just to clarify the nature of the boundaries in native histograms: The boundary that is farther away from zero is inclusive AKA closed, while the boundary that is closer to zero is exclusive AKA open. (Note that this implies that buckets for negative observations are "mirrored" in their behavior.) The zero bucket is a special case, it is closed on both ends.
Also note that the format used in the Prometheus query API is "future proof" for upcoming other bucket schemas. It can describe arbitrary bucket boundaries (in terms of value, openness and closedness).
Ok yes that looks better, array of arrays does make it a bit more compact too. I'd agree, let's try to keep it the same.