logging.googleapis.com/log_entry_count yields out-of-order timestamps

[cinder-fish]

When scraping metrics for this prefix we're seeing timestamps that are reported out-of-order

the timestamp first appears normally, but then seems to start repeating the same period of 1 min over and over until it stops being reported again; (have focused on one series here, but this happens for multiple series under this prefix)

Fri 12 Nov 2021 15:45:04 UTC - [not present]
Fri 12 Nov 2021 15:45:14 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 0 1636731854869
Fri 12 Nov 2021 15:45:25 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 0 1636731865720
Fri 12 Nov 2021 15:45:37 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 1 1636731877555
Fri 12 Nov 2021 15:45:48 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 1 1636731888213
Fri 12 Nov 2021 15:45:58 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 1 1636731898808
Fri 12 Nov 2021 15:46:09 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 1 1636731909404
Fri 12 Nov 2021 15:46:19 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 1 1636731920053
Fri 12 Nov 2021 15:46:30 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 0 1636731930897
Fri 12 Nov 2021 15:46:41 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 0 1636731941582
Fri 12 Nov 2021 15:46:51 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 0 1636731952158
Fri 12 Nov 2021 15:47:02 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 0 1636731962754
Fri 12 Nov 2021 15:47:13 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 0 1636731973352
Fri 12 Nov 2021 15:47:23 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 0 1636731983953
Fri 12 Nov 2021 15:47:34 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 0 1636731994541
Fri 12 Nov 2021 15:47:44 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 0 1636732005151
Fri 12 Nov 2021 15:47:55 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 0 1636732015816
Fri 12 Nov 2021 15:48:06 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 0 1636731966456
Fri 12 Nov 2021 15:48:16 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 0 1636731977097
Fri 12 Nov 2021 15:48:27 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 0 1636731987756
Fri 12 Nov 2021 15:48:38 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 0 1636731998349
Fri 12 Nov 2021 15:48:48 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 0 1636732008940
Fri 12 Nov 2021 15:48:59 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 0 1636732019521
Fri 12 Nov 2021 15:49:09 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 0 1636731970144
Fri 12 Nov 2021 15:49:20 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 0 1636731980742
Fri 12 Nov 2021 15:49:31 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 0 1636731991344
Fri 12 Nov 2021 15:49:41 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 0 1636732002005
Fri 12 Nov 2021 15:49:52 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 0 1636732013088
Fri 12 Nov 2021 15:50:03 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 0 1636731963677
Fri 12 Nov 2021 15:50:14 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 0 1636731974304
Fri 12 Nov 2021 15:50:25 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 0 1636731985262
Fri 12 Nov 2021 15:50:35 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 0 1636731996007
Fri 12 Nov 2021 15:50:46 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 0 1636732006768
Fri 12 Nov 2021 15:50:57 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 0 1636732017408
Fri 12 Nov 2021 15:51:07 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 0 1636731968078
Fri 12 Nov 2021 15:51:18 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 0 1636731978712
Fri 12 Nov 2021 15:51:29 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 0 1636731989356
Fri 12 Nov 2021 15:51:39 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 0 1636731999962
Fri 12 Nov 2021 15:51:50 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 0 1636732010603
Fri 12 Nov 2021 15:52:01 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 0 1636731961217
Fri 12 Nov 2021 15:52:11 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 0 1636731971879
Fri 12 Nov 2021 15:52:22 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 0 1636731982514
Fri 12 Nov 2021 15:52:33 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 0 1636731993254
Fri 12 Nov 2021 15:52:43 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 0 1636732003911
Fri 12 Nov 2021 15:52:54 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 0 1636732014548
Fri 12 Nov 2021 15:53:04 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 0 1636731965157
Fri 12 Nov 2021 15:53:15 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 0 1636731975831
Fri 12 Nov 2021 15:53:26 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 0 1636731986469
Fri 12 Nov 2021 15:53:37 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 0 1636731997860
Fri 12 Nov 2021 15:53:48 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 0 1636732008760
Fri 12 Nov 2021 15:53:59 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 0 1636732019456
Fri 12 Nov 2021 15:54:01 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 0 1636731961719
Fri 12 Nov 2021 15:54:12 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 0 1636731972309
Fri 12 Nov 2021 15:54:22 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 0 1636731982983
Fri 12 Nov 2021 15:54:33 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 0 1636731993651
Fri 12 Nov 2021 15:54:44 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 0 1636732004268
Fri 12 Nov 2021 15:54:54 UTC - stackdriver_cloudsql_database_logging_googleapis_com_log_entry_count{log="cloudsql.googleapis.com/mysql.err",severity="WARNING",unit="1"} 0 1636732015059
Fri 12 Nov 2021 15:55:05 UTC - [not present]
Fri 12 Nov 2021 15:55:16 UTC - [not present]
Fri 12 Nov 2021 15:55:26 UTC - [not present]
Fri 12 Nov 2021 15:55:38 UTC - [not present]

stackdriver_exporter version: 0.11.0

Args:

--monitoring.metrics-offset=60s
--monitoring.metrics-interval=8m
--monitoring.metrics-type-prefixes "logging.googleapis.com/log_entry_count"

Guessing this might be an interaction between the 8m interval and this metric only having a single event in the window above?