Apigee metrics from GCP - bucket type prometheus metrics value is not increasing monotonically

[sundar-ka]

We've been introduced to this stackdriver_exporter recently and started using it to export metrics from GCP infra to our prometheus. Specifically the metrics of Apigee proxy provided as a managed service in GCP. We found a specific type of metric which got exported through the stackdriver exporter is not increasing monotonically in its value over the time. This metric type is used to bucket the range of values in different buckets which then can be used to calculate quantiles using histogram_quantile function with prometheus. Here is an example for metric which is used for calculating latency whose value didn't increase monotonically

Error got from prometheus while calculating histogram quantile using the above metric

This is a helm based deployment in a kubernetes cluster.

Helm Chart version deployed - v4.5.0 Exporter app version deployed - v0.15.0

Expected: The metric _bucket value has to be increasing always like a counter type metric

Help needed . Thanks

[kgeckhart]

👋 can you try again with the latest release of v0.15.1? It contains a bugfix for this very issue.

[sundar-ka]

Thanks. I'll try it and get back

[sundar-ka]

@kgeckhart We've tried v0.15.1 version of the exporter, but still the values are not increasing monotonically. Getting same warning from grafana while calculating histogram quantile. We also use the current latest version of helm chart - v4.5.0

Warning below screenshot

Bucket metric screenshot below ( that doesn't increase monotonically )

kindly help here

[kgeckhart]

Can you check what stackdriver_exporter_build_info says regarding version and revision?

[sundar-ka]

Here is the version and revision details for the exporter from the metric stackdriver_exporter_build_info The version seems 0.15.1

[kgeckhart]

🤔 troubleshooting this isn't easy at this point. I've checked for histograms in my environment which show this behavior and we have none. We also don't have any apigee metrics to go off of to see if it's a specific to something with the metric itself. Do you have other histograms with this behavior or just the apigee one?