ingest_delay=true not working for some metrics

[maxlemieux]

Problem: the exporter doesn't fix metric timestamp for all delayed metrics

Expected: ingest_delay works for all GCP metrics which have ingestDelay metadata available from the GCP API

When using the [gcp/stackdriver exporter], (https://grafana.com/docs/alloy/latest/reference/components/prometheus/prometheus.exporter.gcp/) with ingest_delay=true, at least some of the publisher-delayed metrics still don't appear as expected. Setting ingest_delay=true fixes timestamps for most delayed metrics but not all.

Example affected metric: stackdriver_bigquery_dataset_bigquery_googleapis_com_storage_uploaded_bytes

This seems to be an exporter issue, because the offset time is visible when visiting exporter endpoint directly and checking timestamps. This metric alone has the old timestamp, other metrics behave as expected.

This metric has the expected notice in GCP docs: "After sampling, data is not visible for up to 21720 seconds"

I also verified that the metric has this metadata and should be respected by the exporter. Here's the metric descriptor:

# List all BigQuery metric descriptors
curl -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     "https://monitoring.googleapis.com/v3/projects/my-project-id/metricDescriptors?filter=metric.type%3Dstarts_with(%22bigquery.googleapis.com%22)"

    {
      "name": "projects/my-project/metricDescriptors/bigquery.googleapis.com/storage/uploaded_bytes",
      "labels": [
        {
          "key": "api",
          "description": "Api used to upload the data (batch import or streaming)"
        },
        {
          "key": "table",
          "description": "Table name."
        }
      ],
      "metricKind": "DELTA",
      "valueType": "INT64",
      "unit": "By",
      "description": "Uploaded bytes.",
      "displayName": "Uploaded bytes",
      "type": "bigquery.googleapis.com/storage/uploaded_bytes",
      "metadata": {
        "launchStage": "GA",
        "samplePeriod": "60s",
        "ingestDelay": "21720s",
        "timeSeriesResourceHierarchyLevel": [
          "PROJECT"
        ]
      },
      "launchStage": "GA",
      "monitoredResourceTypes": [
        "bigquery_dataset"
      ]
    },

However, despite the presence of this metadata and setting ingest_delay=true, the metric is not scraped by the collector because its timestamp is always 6 hours old in the GCP exporter.

Workaround: in the Alloy collector, scrape the exporter twice, use honor_timestamps to further control the timestamp to set it to current time for this one metric. This works for now:

prometheus.scrape "test_exporter_main" {
  targets = [
    {"__address__" = "localhost:8081"},
  ]
  honor_timestamps = true
  forward_to = [prometheus.relabel.filter_main.receiver]
}

prometheus.scrape "test_exporter_lagging" {
  targets = [
    {"__address__" = "localhost:8081"},
  ]
  honor_timestamps = false
  forward_to = [prometheus.relabel.filter_lagging.receiver]
}

prometheus.relabel "filter_main" {
  rule {
    source_labels = ["__name__"]
    regex = "old_metric_with_past_timestamp"
    action = "drop"
  }
  forward_to = [prometheus.remote_write.metrics_service.receiver]
}

prometheus.relabel "filter_lagging" {
  rule {
    source_labels = ["__name__"]
    regex = "old_metric_with_past_timestamp"
    action = "keep"
  }
  forward_to = [prometheus.remote_write.metrics_service.receiver]
}

Please let me know if I can provide more information about this issue. Thank you!