Keda Auto Scaling failing with Pub/Sub

[slv-306]

Report

Hi Team Here is the error which I'm facing while autoscaling using GCP/Pub-Sub with Keda. AbleToScale True SucceededGetScale the HPA controller was able to get the target's current scale ScalingActive False FailedGetExternalMetric the HPA was unable to compute the replica count: unable to get external metric default/s0-gcp-ps-echo-read/&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: pubsub-scaledobject,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: no matching metrics found for s0-gcp-ps-echo-read Events: Type Reason Age From Message

---

Warning FailedGetExternalMetric 92s (x12 over 4m25s) horizontal-pod-autoscaler unable to get external metric default/s0-gcp-ps-echo-read/&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: pubsub-scaledobject,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: no matching metrics found for s0-gcp-ps-echo-read Warning FailedComputeMetricsReplicas 76s (x13 over 4m25s) horizontal-pod-autoscaler failed to compute desired number of replicas based on listed metrics for Deployment/default/pubsub: invalid metrics (1 invalid out of 1), first error is: failed to get s0-gcp-ps-echo-read external metric: unable to get external metric default/s0-gcp-ps-echo-read/&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: pubsub-scaledobject,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: no matching metrics found for s0-gcp-ps-echo-read

Expected Behavior

Even when I'm checking metrics it isn't showing metrics kubectl get --raw '/apis/external.metrics.k8s.io/v1beta1/namespaces/api/default/s0-gcp-ps-echo-read'

Error from server (NotFound): the server could not find the requested resource

Actual Behavior

Type Reason Age From Message

---

Warning FailedGetExternalMetric 3m58s (x467 over 129m) horizontal-pod-autoscaler unable to get external metric default/s0-gcp-ps-echo-read/&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: pubsub-scaledobject,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: no matching metrics found for s0-gcp-ps-echo-read

Steps to Reproduce the Problem

HPA is failing

Logs from KEDA operator

1.6517362746580615e+09 ERROR scalehandler Error getting scale decision {"scaledobject.Name": "pubsub-scaledobject", "scaledObject.Namespace": "default", "scaleTarget.Name": "pubsub", "error": "unexpected end of JSON input"} github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers /workspace/pkg/scaling/scale_handler.go:278 github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop /workspace/pkg/scaling/scale_handler.go:149

KEDA Version

2.6.1

Kubernetes Version

1.20

Platform

No response

Scaler Details

GCP- Pub/Sub

Anything else?

No response

[tomkerkhove]

Since this is for GCP, did you make sure that it is reachable?

Why does Google Kubernetes Engine (GKE) 1.16 fail to fetch external metrics?

See our troubleshooting guide: https://keda.sh/docs/2.7/troubleshooting/

[mhoppe-cedar]

I saw a similar error when my keda metrics server did not have proper IAM permissions associated with it. Went back to check the keda metrics server deployment, the keda operator deployment, the scaledobject, and the keda operator serviceaccount roles/permissions. After ensuring they all agreed and were sufficient, it worked.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed due to inactivity.

[rejuljames]

@mhoppe-cedar Can you give me permission you have given for the your service account ?

[mau21mau]

The service account must have roles/monitoring.viewer role