kubectl get --raw /apis/external.metrics.k8s.io/v1beta1 returns Error ServiceUnavailable

[eyearian]

Report

I currently have KEDA installed via raw manifests - or helm charts that have been kubesplit into raw manifests. I am converting this to be installed using helm charts via argocd. I am matching the same version - 2.11.2 to enable argo to manage this app. Additionally I have totally ripped out the current KEDA install and have done a fresh install of KEDA but getting the same api service error.

I have seen the other keda issues relating to dummy scaler objects, but this should be fixed in version 2.11.2 and unrelated to this issue.

Expected Behavior

Running this command - kubectl get --raw /apis/external.metrics.k8s.io/v1beta1 | jq . on clusters with KEDA installed via raw manifests (no argocd or helm install) currently this:

  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "external.metrics.k8s.io/v1beta1",
  "resources": [
    {
      "name": "externalmetrics",
      "singularName": "",
      "namespaced": true,
      "kind": "ExternalMetricValueList",
      "verbs": [
        "get"
      ]
    }
  ]
}

Actual Behavior

When I run kubectl get --raw /apis/external.metrics.k8s.io/v1beta1 | jq . on an EKS cluster with the helm chart based KEDA with version 2.11.1 the following is returned: Error from server (ServiceUnavailable): the server is currently unable to handle the request

API service details:

Name:         v1beta1.external.metrics.k8s.io
Namespace:
Labels:       app.kubernetes.io/component=operator
              app.kubernetes.io/instance=keda
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=v1beta1.external.metrics.k8s.io
              app.kubernetes.io/part-of=keda-operator
              app.kubernetes.io/version=2.11.2
              argocd.argoproj.io/instance=test-keda
              helm.sh/chart=keda-2.11.2
Annotations:  <none>
API Version:  apiregistration.k8s.io/v1
Kind:         APIService
Metadata:
  Creation Timestamp:  2024-04-23T14:05:34Z
  Resource Version:    721980299
  UID:                 7bbaa574-28a1-4055-abb9-48b686dedcaf
Spec:
  Ca Bundle:               LS0tL...
  Group:                   external.metrics.k8s.io
  Group Priority Minimum:  100
  Service:
    Name:            keda-operator-metrics-apiserver
    Namespace:       keda
    Port:            443
  Version:           v1beta1
  Version Priority:  100
Status:
  Conditions:
    Last Transition Time:  2024-04-23T14:05:34Z
    Message:               all checks passed
    Reason:                Passed
    Status:                True
    Type:                  Available
Events:                    <none>

Steps to Reproduce the Problem

1. Install keda via kubesplit helm charts
2. Use the helm chart package to have helm/argo manage the chart

Logs from KEDA operator

Metrics API server: "Connection to KEDA Metrics Service gRPC server has been successfully established"

Operator:

Ensuring CA cert	{"name": "keda-admission", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration", "name": "keda-admission", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration"}
INFO	cert-rotation	Ensuring CA cert	{"name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService", "name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService"}
INFO	cert-rotation	CA certs are injected to webhooks
INFO	grpc_server	Starting Metrics Service gRPC Server	{"address": ":9666"}

KEDA Version

2.11.2

Kubernetes Version

1.26

Platform

Amazon Web Services

Scaler Details

No response

Anything else?

I'm unsure if this is even an issue, looking for confirmation on whether this is a bug or something is wrong as I cant get the raw version of the external.metrics api service.

[JorTurFer]

I missed this issue, sorry :(

Could you try upgrading KEDA to a greater version?