Stackdriver metrics timestamp sometimes goes backwards in time

[mplzik]

stackdriver-exporter 0.9.0 introduced timestamps associated with each exported series. When doing experiments, in some cases (in our case stackdriver_l_7_lb_rule_logging_googleapis_com-metrics) the timestamp jumped as much as 2 minutes into the past (and return back to the present). When fetching the metrics every ~2seconds, the timestamp differences were as follows:

2020-06-16T17:15:59+02:00  2084
2020-06-16T17:16:01+02:00  -57920
2020-06-16T17:16:03+02:00  1878
2020-06-16T17:16:05+02:00  2023
2020-06-16T17:16:07+02:00  1972
2020-06-16T17:16:09+02:00  1941
2020-06-16T17:16:11+02:00  1940
2020-06-16T17:16:13+02:00  62148
2020-06-16T17:16:15+02:00  1954
2020-06-16T17:16:17+02:00  2026
2020-06-16T17:16:19+02:00  1904
2020-06-16T17:16:21+02:00  2252
2020-06-16T17:16:23+02:00  1924

This doesn't work well with Prometheus, since any data that appear from the past will get dropped.

[SuperQ]

Thous sounds like a bit of a bug in the stackdriver API results. I wonder if this is somewhat "working as intended", as stackdriver might be returning old data as well as just the old timestamp. Ignoring these stale stackdriver results is probably a good thing.

[mplzik]

It would be still nice to see whether there's any way to handle this in a more correct manner -- in a way, this workaround basically means that the data would get downsampled to 1m or 2m sample rate at random moments and might not be suitable for range queries with 2m range or less.

[SuperQ]

From what I can tell, this is the correct behavior. Stackdriver is returning stale/incorrect data and Prometheus needs to ignore it. Otherwise it would be ingesting data that is duplicate of past events. This is a Stackdriver bug.

[mplzik]

A small update -- looks like this issue has been triggered by custom metrics only.