GCP Stackdriver scaler doesn't work with centralized metrics scope project

[mjneth]

Report

The GCP Stackdriver scaler is unable to pull metrics from a centralized metrics scope project because it is hardcoded to add the wrong project ID. It was updated as part of issue #5256 and PR #5258 to add a project_id filter that is the project being queried but this breaks the query if you're pulling from a metrics scope project and trying to filter the query by a different project that is included in the metrics scope.

The changes in #5258 work if you want to query a metric that came from the same project your trigger is set to query from but that doesn't work if you use a separate, centralized metrics scope project.

Expected Behavior

The scaler can successfully pull metrics from a centralized metrics scope project.

Actual Behavior

The scaler can't pull metrics because the filter is hardcoded to add an incorrect resource label to the query.

Steps to Reproduce the Problem

1. Create a project, projectA. This is the project that has Keda running in it.
2. Create a second project, projectB. This is the project where the intended metric is published. This should be a custom metric that is only published in this project and not the others (published via something like prometheus or otelcollector).
3. Create a third project, projectC. This is the project that has the metric scope set to include projectA and projectB.
4. In projectA, create a ScaledObject with a trigger that has type of gcp-stackdriver, projectId of projectC, and a filter that queries the metric published in projectB.

Logs from KEDA operator

My filter is: filter: 'metric.type="prometheus.googleapis.com/our_custom_metric_name/gauge" AND metric.label.store_id="a.unique.id"'

2024-05-08T20:41:12Z    INFO    fallback        Suppressing error, falling back to fallback.replicas    {"scaledObject.Namespace": "mntest", "scaledObject.Name": "gcp-stackdriver-scaledobject", "suppressedError": "could not find stackdriver metric with filter metric.type=\"prometheus.googleapis.com/our_custom_metric_name/gauge\" AND metric.label.store_id=\"a.unique.id\" AND resource.labels.project_id=\"projectC\"", "fallback.replicas": 2}

Note that it's automatically adding AND resource.labels.project_id=\"projectC\"" to the query that's not present in our filter which is causing it to fail.

KEDA Version

2.14.0

Kubernetes Version

1.27

Platform

Google Cloud

Scaler Details

GCP Stackdriver

Anything else?

No response

[zroubalik]

@mjneth thanks for reporting, are you willing to contribute a fix that will work for all solutions?

[mjneth]

@mjneth thanks for reporting, are you willing to contribute a fix that will work for all solutions?

If I can find the time soon to dig deeper then i'll try. I think the solution is not include the hardcoded project ID when using the gcp-stackdriver scaler since people can add project_id to the filter query to match whatever they need to do.