Track delta metrics between scrapes

[kgeckhart]

This PR introduces support for aggregating DELTA metrics in-memory. This implementation is somewhat based on https://github.com/prometheus-community/stackdriver_exporter/pull/167 but with extra functionality which is intended to solve the problem identified here, https://github.com/prometheus-community/stackdriver_exporter/pull/167#issuecomment-1217127124.

I chose to inject interfaces for the delta and distribution stores at the Collector level to try to keep library friendliness intact, https://github.com/prometheus-community/stackdriver_exporter/pull/157

I did as much as I could to explain the solution and the tradeoffs with aggregating deltas in this manner in the README.md. The bulk of the changes are in monitoring_metrics.go where I tried to obfuscate as much of the delta logic as possible.

Should hopefully resolve #116 and #25

[SuperQ]

Thanks, I'll take a look at this soon.

To start, this needs a DCO sign-off. You can use git commit -s --amend to add it.

[ahjmorton]

Hello @SuperQ / @kgeckhart, this looks like it solves a really important issue with the exporter where DELTA metrics seem pretty much unusable in certain cases.

Can I ask what is stopping it from being merged and if there is anything I can do to speed that up? This issue is making monitoring of some of our GCP dependencies a lot harder as the data is split into 2 places.

[kgeckhart]

Hi @ahjmorton, I'll defer to @SuperQ for the review + merge side of things. I can say we have been running this version in production for a little over three weeks. Our largest exporter is tracking ~500k metrics without any noticeable issues.

[SuperQ]

CC @igorwwwwwwwwwwwwwwwwwwww, this may interest you.

[igorwwwwwwwwwwwwwwwwwwww]

cc @t4cc0re