Optionally handle DELTA metrics as counters

[osabina]

I've cleaned up the internal set of commits we are (successfully, as far as we can determine) running as of this week (8/8/2022) and it seemed to make sense to drop this as a PR instead of a random branch floating out there.

Open to whatever cleanup/changes/suggestions folks have in order to get this merged into the mainline.

[kgeckhart]

I figured it might be easier to move this convo to the PR. Related to question 3 from here, I think this might be an issue for slower moving metrics. Everything works really well when the metric is constantly changing which is awesome! Here's an example with a slower moving metric, Top graph is a 5m rate calculation and bottom is the raw counter. Seems as though GCP will produce a 0 for a delta for ~5 minutes before dropping the metric. When the metric starts reporting again the rate doesn't start reporting because of the gap in data. The rate only starts reporting on the second change which probably isn't ideal.

I'm not quite sure how to resolve this without going to the solution which always reports the counters even if GCP drops it for a longer period of time. Happy to try to merge your work with mine which attempts to solve the problem this way. I'm currently attempting to fix an issue with the implementation which moves the reported time too far in to the future.

[osabina]

@kgeckhart thanks for the excellent explanation of the problem -- we must only be tracking things that are getting reasonable traffic since everyone seems happy (so far) with what we have. But, certainly If you can integrate a solution to this, that would be awesome!

Perhaps have a flag that sets the TTL for continuing to export metrics no longer showing up, default 0s -- then folks could adjust for their needs?

[osabina]

Sorry meant to close this since @kgeckhart got this fix into the mainline already!