Setting 30s metrics interval yields empty dataflow metrics

[calonso]

Setting STACKDRIVER_EXPORTER_MONITORING_METRICS_INTERVAL to 30s and adding dataflow.googleapis.com/job to the list of STACKDRIVER_EXPORTER_MONITORING_METRICS_TYPE_PREFIXES brings Dataflow related metrics empty

[frodenas]

This happens with some metrics because they have a higher latency (see https://github.com/frodenas/stackdriver_exporter/issues/3 or https://github.com/frodenas/stackdriver_exporter/issues/7). Have you tried increasing the monitoring.metrics-interval?

[calonso]

Yeah, I've tried setting it to 1m or even 5m and some of them come, but specifically bigquery.googleapis.com prefix is empty. Increasing log level seems to at least query them, though...

INFO[0000] Starting stackdriver_exporter (version=, branch=, revision=)  source="stackdriver_exporter.go:83"
INFO[0000] Build context (go=go1.9.2, user=, date=)      source="stackdriver_exporter.go:84"
INFO[0000] Listening on :9255                            source="stackdriver_exporter.go:110"
DEBU[0003] Listing Google Stackdriver Monitoring metric descriptors starting with `bigquery.googleapis.com`...  source="monitoring_collector.go:212"
DEBU[0003] Retrieving Google Stackdriver Monitoring metrics for descriptor `bigquery.googleapis.com/query/count`...  source="monitoring_collector.go:166"
DEBU[0003] Retrieving Google Stackdriver Monitoring metrics for descriptor `bigquery.googleapis.com/query/scanned_bytes`...  source="monitoring_collector.go:166"
DEBU[0003] Retrieving Google Stackdriver Monitoring metrics for descriptor `bigquery.googleapis.com/slots/total_allocated_for_reservation`...  source="monitoring_collector.go:166"
DEBU[0003] Retrieving Google Stackdriver Monitoring metrics for descriptor `bigquery.googleapis.com/slots/total_available`...  source="monitoring_collector.go:166"
DEBU[0003] Retrieving Google Stackdriver Monitoring metrics for descriptor `bigquery.googleapis.com/storage/stored_bytes`...  source="monitoring_collector.go:166"
DEBU[0003] Retrieving Google Stackdriver Monitoring metrics for descriptor `bigquery.googleapis.com/query/scanned_bytes_billed`...  source="monitoring_collector.go:166"
DEBU[0003] Retrieving Google Stackdriver Monitoring metrics for descriptor `bigquery.googleapis.com/storage/uploaded_bytes`...  source="monitoring_collector.go:166"
DEBU[0003] Retrieving Google Stackdriver Monitoring metrics for descriptor `bigquery.googleapis.com/slots/allocated_for_project`...  source="monitoring_collector.go:166"
DEBU[0003] Retrieving Google Stackdriver Monitoring metrics for descriptor `bigquery.googleapis.com/storage/uploaded_row_count`...  source="monitoring_collector.go:166"
DEBU[0003] Retrieving Google Stackdriver Monitoring metrics for descriptor `bigquery.googleapis.com/storage/table_count`...  source="monitoring_collector.go:166"
DEBU[0003] Retrieving Google Stackdriver Monitoring metrics for descriptor `bigquery.googleapis.com/slots/allocated_for_reservation`...  source="monitoring_collector.go:166"
DEBU[0003] Retrieving Google Stackdriver Monitoring metrics for descriptor `bigquery.googleapis.com/storage/uploaded_bytes_billed`...  source="monitoring_collector.go:166"
DEBU[0003] Retrieving Google Stackdriver Monitoring metrics for descriptor `bigquery.googleapis.com/query/execution_times`...  source="monitoring_collector.go:166"
DEBU[0003] Retrieving Google Stackdriver Monitoring metrics for descriptor `bigquery.googleapis.com/slots/allocated_for_project_and_job_type`...  source="monitoring_collector.go:166"

[jameshartig]

@calonso see my explanation in #33. BigQuery metrics can be as backed up as 6 hours.

[agusl88]

maybe being able to customize the monitoring.metrics-interval for different services could be a solution?

[jameshartig]

You should be able to set the interval to something like 6 hours and it won't break the other services because it only uses the latest metrics, so you'll just be pulling more data than necessary but it shouldn't break the metrics (as far as I know). I'd try that and see, you'll just need to note somewhere that the bigquery ones are from 6 hours ago and they're not current, though they'll look like their current from Prometheus's point of view.

[agusl88]

I increased the metrics interval to 360m (6 hours) and i'm still getting nothing from bigquery and storage metrics.

[dcaba]

A couple of notes about this:

• Setting longer intervals works for me for BigQuery metrics (it'd be great Google documents these behaviors / lags we can expect when querying stackdriver... I didn't find any note), so I'd close this issue.
• But, yeah, I'm afraid the exporter is "dumb" and tries to collect all metrics for a given interval, regardless its only going to offer the newest metric to Prometheus. It'd be great if it stops querying stackdriver api once it gets the latest value in the given interval. So, ...
• ... with the current behavior, a customizable interval (and maybe offset) per prefix would be very welcome, to reduce the number of requests against stackdriver API (which impact costs)

The second and third point could be part of a new feature request. But the original issue is no longer a concern once for the exporter once you know how metric behaves for some specific google services.