Allow multiple gauge values within the same MetricAggregator instance.

[brandondahler]

Adds the provided timestamp to the metric instance's context. Use of multiple gauge values for a single metric requires the timestamps to be passed explicitly.

Problem:
Currently calling self.gauge() with the same metric name, hostname, and tags but different timestamp only retains the last call's value.

Use case: Upstream API that provides data has a variable delay before values are available. When data is available, it provides multiple values for different timestamps all at once.

Proposed solution: Includes the provided timestamp as part of the metric instance's context to allow multiple instances that differ only by timestamp.

Notes about solution: This technically affects all metrics, potentially changing the way the previous metric instance is selected. The updated code only affects previous implementations if and only if the timestamp parameter is specified. The timestamp parameter is currently only used by the gauge metric.

[brandondahler]

Checks failing due to change in how MetricAggregator handles multiple calls to self.gauge when timestamp parameter is different (as intended for this request).

[degemer]

Hey @brandondahler, thanks for the PR!

This could be a pretty big change, could you elaborate on your use case ? I.e. how far in the past are the timestamps you are adding ? What's the resolution (one metric every second, 10 seconds, ...) ?

Let me know if you have any question!

[brandondahler]

This PR is a product of what I had to do in implementation of AzureServiceBusCheck.py:40-53 (which I plan on submitting as a community check eventually).

Specifically the management_service.get_metrics_data_queue(...) function calls an Azure web service. That web service returns aggregated metrics data (such as messages outgoing and incoming) over a configurable time period, I'm using the smallest at 5 minutes.

The problem is that the metrics aren't immediately available every 5 minutes, instead they seem to slowly come in based on load of their system, usually within 15 minutes (or never if the value is 0). The service is structured so that you can query time ranges and it will return all non-zero results within that time range.

Ideally I'd be able to query this service every 5 minutes and get only the new metric values, even if there are multiple available since the last time I tried, and submit all of them at once. However, since the metrics only differ by value and timestamp, I can only submit a single value for that run.

I do understand that there are other ways for me to solve this particular problem, I'm submitting this PR because the functionality seems confusing.