stackdriver_exporter
stackdriver_exporter copied to clipboard
Error on ingesting out-of-order samples
for the last 5+ days, I'm getting via Prometheus the error
component="scrape manager" scrape_pool=serviceMonitor/monitoring/prometheus-stackdriver-exporter/0 target=http://10.205.6.116:9255/metrics msg="Error on ingesting out-of-order samples" num_dropped=946
I tried playing with the offsets and intervals, but I cannot fix this error,
the error I am getting in alertmanager is PrometheusDuplicateTimestamps
and I'm using chart version 4.4.1 (I upgraded it after the alert started showing up)
here are the values I pass to configure this exporter
stackdriver:
projectId: '{{values.projectId}}'
metrics:
typePrefixes: >-
logging.googleapis.com/user,
pubsub.googleapis.com,
bigtable.googleapis.com,
cloudsql.googleapis.com,
custom.googleapis.com,
dataflow.googleapis.com,
monitoring.googleapis.com/uptime_check,
serviceruntime.googleapis.com/quota,
file.googleapis.com,
loadbalancing.googleapis.com
resources:
requests:
cpu: 100m
memory: 128Mi
serviceMonitor:
enabled: true
interval: 20s
additionalLabels:
release: prometheus
and some time I get this in the logs
ts=2023-12-27T17:13:05.253Z caller=monitoring_collector.go:315 level=error msg="error retrieving Time Series metrics for descriptor" descriptor=cloudsql.googleapis.com/database/mysql/innodb_buffer_pool_pages_dirty err="Get \"https://monit
oring.googleapis.com/v3/projects/fakeproject/timeSeries?alt=json&filter=metric.type%3D%22cloudsql.googleapis.com%2Fdatabase%2Fmysql%2Finnodb_buffer_pool_pages_dirty%22&interval.endTime=2023-12-27T17%3A13%3A05.233788804Z&interval.startTime=20
23-12-27T17%3A08%3A05.233788804Z&prettyPrint=false\": EOF"
ts=2023-12-27T17:13:05.253Z caller=monitoring_collector.go:315 level=error msg="error retrieving Time Series metrics for descriptor" descriptor=cloudsql.googleapis.com/database/mysql/open_table_definitions err="Get \"https://monitoring.go
ogleapis.com/v3/projects/fakeproject/timeSeries?alt=json&filter=metric.type%3D%22cloudsql.googleapis.com%2Fdatabase%2Fmysql%2Fopen_table_definitions%22&interval.endTime=2023-12-27T17%3A13%3A05.233788804Z&interval.startTime=2023-12-27T17%3A08
%3A05.233788804Z&prettyPrint=false\": EOF"
ts=2023-12-27T17:13:05.253Z caller=monitoring_collector.go:315 level=error msg="error retrieving Time Series metrics for descriptor" descriptor=cloudsql.googleapis.com/database/sqlserver/transactions/forwarded_record_count err="Get \"http
s://monitoring.googleapis.com/v3/projects/fakeproject/timeSeries?alt=json&filter=metric.type%3D%22cloudsql.googleapis.com%2Fdatabase%2Fsqlserver%2Ftransactions%2Fforwarded_record_count%22&interval.endTime=2023-12-27T17%3A13%3A05.233788804Z&i
nterval.startTime=2023-12-27T17%3A08%3A05.233788804Z&prettyPrint=false\": EOF"
ts=2023-12-27T17:13:05.254Z caller=monitoring_collector.go:315 level=error msg="error retrieving Time Series metrics for descriptor" descriptor=cloudsql.googleapis.com/database/replication/log_archive_failure_count err="Get \"https://moni
toring.googleapis.com/v3/projects/fakeproject/timeSeries?alt=json&filter=metric.type%3D%22cloudsql.googleapis.com%2Fdatabase%2Freplication%2Flog_archive_failure_count%22&interval.endTime=2023-12-27T17%3A13%3A05.233788804Z&interval.startTime=
2023-12-27T17%3A08%3A05.233788804Z&prettyPrint=false\": EOF"
ts=2023-12-27T17:13:05.254Z caller=monitoring_collector.go:315 level=error msg="error retrieving Time Series metrics for descriptor" descriptor=cloudsql.googleapis.com/database/postgresql/insights/perquery/lock_time err="Get \"https://mon
itoring.googleapis.com/v3/projects/fakeproject/timeSeries?alt=json&filter=metric.type%3D%22cloudsql.googleapis.com%2Fdatabase%2Fpostgresql%2Finsights%2Fperquery%2Flock_time%22&interval.endTime=2023-12-27T17%3A13%3A05.233788804Z&interval.star
tTime=2023-12-27T17%3A08%3A05.233788804Z&prettyPrint=false\": EOF"
ts=2023-12-27T17:13:05.693Z caller=monitoring_collector.go:228 level=error msg="Error while getting Google Stackdriver Monitoring metrics" err="Get \"https://monitoring.googleapis.com/v3/projects/fakeproject/timeSeries?alt=json&filter=metric
.type%3D%22cloudsql.googleapis.com%2Fdatabase%2Fmysql%2Finnodb_buffer_pool_pages_dirty%22&interval.endTime=2023-12-27T17%3A13%3A05.233788804Z&interval.startTime=2023-12-27T17%3A08%3A05.233788804Z&prettyPrint=false\": EOF"