stackdriver-prometheus-sidecar
stackdriver-prometheus-sidecar copied to clipboard
keys show up, but no data
Trying to load any metrics into metrics explorer shows a "no data for this time range" in the graph. The metrics are being pushed as I see metrics ingress bytes count is >0. Is there some way to debug the metrics actually being processed and viewable in metrics explorer? One special thing to note, I do not use GKE. I have to untoggle 'show active' to find these metrics keys.
Here's my prometheus config
global:
scrape_interval: 15s
evaluation_interval: 30s
# scrape_timeout is set to the global default (10s).
external_labels:
monitor: codelab
foo: bar
scrape_configs:
- job_name: thegraph
metrics_path: /metrics
static_configs:
- targets:
- thegraph:8040
labels:
_generic_location: us-central1-a
_generic_namespace: thegraph
- job_name: stackdriver_sidecar
metrics_path: /metrics
static_configs:
- targets:
- stackdriver:9091
labels:
_generic_location: us-central1-a
_generic_namespace: stackdriver
I am seeing the same issue on a GKE cluster. I see a list of inactive metrics in Cloud Monitoring, but neither of them contain any data.
Relevant Prometheus config (metrics show up in Prometheus):
scrape_configs:
- job_name: serviceMonitor/ingress-nginx/ingress-nginx-controller/0
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 10s
Versions used:
- Stackdriver Prometheus Sidecar - gcr.io/stackdriver-prometheus/stackdriver-prometheus-sidecar:0.10.0
- Prometheus - Tested with 2.30 and 2.19.0 (to match with compatibility matrix in the README)
Args passed to Stackdriver Prometheus Sidecar:
"--stackdriver.project-id=${var.gcp_project_id}",
"--prometheus.wal-directory=${var.data_dir}/wal",
"--stackdriver.kubernetes.location=${var.gke_region}",
"--stackdriver.kubernetes.cluster-name=${var.gke_cluster_name}",
Stackdriver Prometheus Sidcar logs output:
│ level=warn ts=2021-11-12T10:10:28.769Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = Field timeSeries[35].points[0].distributionValue had an invalid value: Distribution |explicit_buckets.bounds| does not have at least one entry." │
│ level=warn ts=2021-11-12T10:10:58.664Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = Field timeSeries[35].points[0].distributionValue had an invalid value: Distribution |explicit_buckets.bounds| does not have at least one entry." │
│ level=warn ts=2021-11-12T10:11:28.688Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = Field timeSeries[35].points[0].distributionValue had an invalid value: Distribution |explicit_buckets.bounds| does not have at least one entry." │
│ level=warn ts=2021-11-12T10:11:58.785Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = Field timeSeries[35].points[0].distributionValue had an invalid value: Distribution |explicit_buckets.bounds| does not have at least one entry." │
│ level=warn ts=2021-11-12T10:12:28.593Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = Field timeSeries[35].points[0].distributionValue had an invalid value: Distribution |explicit_buckets.bounds| does not have at least one entry." │
│ level=warn ts=2021-11-12T10:12:58.701Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = Field timeSeries[35].points[0].distributionValue had an invalid value: Distribution |explicit_buckets.bounds| does not have at least one entry." │
│ level=warn ts=2021-11-12T10:13:28.748Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = Field timeSeries[35].points[0].distributionValue had an invalid value: Distribution |explicit_buckets.bounds| does not have at least one entry." │
│ level=warn ts=2021-11-12T10:13:58.582Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = Field timeSeries[35].points[0].distributionValue had an invalid value: Distribution |explicit_buckets.bounds| does not have at least one entry." │
│ level=warn ts=2021-11-12T10:14:28.577Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = Field timeSeries[35].points[0].distributionValue had an invalid value: Distribution |explicit_buckets.bounds| does not have at least one entry." │
│ level=warn ts=2021-11-12T10:14:58.684Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = Field timeSeries[35].points[0].distributionValue had an invalid value: Distribution |explicit_buckets.bounds| does not have at least one entry." │
│ level=warn ts=2021-11-12T10:15:28.754Z caller=queue_manager.go:534 component=queue_manager msg="Unrecoverable error sending samples to remote storage" err="rpc error: code = InvalidArgument desc = Field timeSeries[35].points[0].distributionValue had an invalid value: Distribution |explicit_buckets.bounds| does not have at least one entry."