Keda failed to fetch PubSub metrics randomly

[chenele]

Report

Hi

We are using Keda 2.5.0 with GKE.

There is some kind of bug when using GCP PubSub with Keda, while from time to time it failed to fetch metrics from stackdriver and says the metric doesnt exist, although few minutes later all good.

anyone encounter this? any workaround? 10x

Expected Behavior

fetch metrics all the time

Actual Behavior

failed to fetch metrics randomly , but too frequent

Steps to Reproduce the Problem

Logs from KEDA operator

2022-06-12T05:45:22.762Z	ERROR	gcp_pub_sub_scaler	error getting Active Status	{"error": "could not find stackdriver metric with filter metric.type=\"pubsub.googleapis.com/subscription/num_undelivered_messages\" AND resource.labels.subscription_id=\"my-subscription\""}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
	/workspace/pkg/scaling/scale_handler.go:261
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
	/workspace/pkg/scaling/scale_handler.go:146

KEDA Version

2.5.0

Kubernetes Version

1.21

Platform

Google Cloud

Scaler Details

PubSub

Anything else?

No response

[JorTurFer]

hey @chenele There were problems related with the internal cache in KEDA v2.5, please update it to at least v2.6.1 (or to latest v2.7.1) let me know if the problem persists after updating KEDA

[zroubalik]

yeah, go straight to 2.7.1, there were also some improvements in GCP scalers.

[chenele]

Cool, 10x. we will update to 2.7.1 and see if this issue remains.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed due to inactivity.

[prakharjoshi]

Hi @zroubalik , we are running keda 2.7.1 version and using the stack driver scaler to pull load-balancer metrics for scaling applications, we are still getting intermediate errors for fetching metrics

1 logr.go:279] keda_metrics_adapter/gcp_stackdriver_scaler "msg"="error getting metric value" "error"="could not find stackdriver metric with filter metric.type=\"loadbalancing.googleapis.com/https/backend_request_count\" AND resource.type=\"https_lb_rule\" AND resource.label.backend_target_name=\"<application_name>\""  ```

[JorTurFer]

Could you open a new issue to track this please @prakharjoshi ?

[prakharjoshi]

@JorTurFer Created a new issue for your reference https://github.com/kedacore/keda/issues/3937