AKS icon indicating copy to clipboard operation
AKS copied to clipboard

support for Kubernetes Event-driven Autoscaling workload identity

Open JonasCordsen opened this issue 2 years ago • 15 comments

Describe the solution you'd like it would be great to support use of workload identity (With a managed identity) in KEDA in order to handle the secrets that is used by trigger

IE guide form keda https://keda.sh/docs/2.9/authentication-providers/azure-key-vault/

Describe alternatives you've considered I have considered to setup KEDA by hand, but the support for this would make it a lot easier

JonasCordsen avatar Mar 08 '23 09:03 JonasCordsen

KEDA 2.10 add-on deployed within AKS 1.26 supports workload identity but it requires extra setup

  • Create a Managed Identity with the corresponding permissions based on the external azure resources you want to access (Keyvault, Azure Monitor Workspace, ...) and save its Client ID for later
  • Get OIDC issuer URL of your cluster: export AKS_OIDC_ISSUER="$(az aks show -n "<AKS cluster name>" -g "<RGP name>" --query "oidcIssuerProfile.issuerUrl" -otsv)"
  • Create federated credentials: az identity federated-credential create --name "<Federated Credential name>" --identity-name "<Managed Id name>" --resource-group "<RGP name>" --issuer "${AKS_OIDC_ISSUER}" --subject system:serviceaccount:kube-system:keda-operator --audience api://AzureADTokenExchange
  • ~~Add Managed Identity client ID config in the keda-operator's service account in kube-system namespace:~~ NOT NEEDED per @kratkyzobak's comment
apiVersion: v1
automountServiceAccountToken: true
kind: ServiceAccount
metadata:
  annotations:
    azure.workload.identity/client-id: <Managed ID Client ID>
  • ~~Restart the keda-metrics-apiserver pods~~

  • ~~keda-metrics-apiserver pods should have an environment variable AZURE_CLIENT_ID set with the value you set in the service account~~

  • Add the User Managed Identity Client ID to the

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: trigger-authentication-name
  namespace: default # must be same namespace as the ScaledObject
spec:
  podIdentity:
   provider: azure-workload
    identityId: <Managed ID Client ID>

~~I'm honestly not sure about overriding the service account manifest with the Managed Id's client ID, not sure it's future proof in terms of AKS upgrades. I would be nice if the AKS add-on provisioning was taking care about the Managed Id provisioning + federated credentials for us, or at least having CLI option to give the client ID ourselves, or even better support for both solutions.~~

EppO avatar Apr 28 '23 19:04 EppO

azure.workload.identity/client-id annotation should not be required to modify. You should specify clientId in podIdentity.identityId for each specific TriggerAuthentication resource.

kratkyzobak avatar Jul 12 '23 12:07 kratkyzobak

that's awesome! no need to do this ugly hack then, thanks for the tip!! I edited my original comment for the updated recipe

EppO avatar Jul 13 '23 01:07 EppO

Action required from @Azure/aks-pm

Issue needing attention of @Azure/aks-leads

Issue needing attention of @Azure/aks-leads

Issue needing attention of @Azure/aks-leads

Issue needing attention of @Azure/aks-leads

Issue needing attention of @Azure/aks-leads

Issue needing attention of @Azure/aks-leads

Issue needing attention of @Azure/aks-leads

Issue needing attention of @Azure/aks-leads

Issue needing attention of @Azure/aks-leads

Issue needing attention of @Azure/aks-leads

Issue needing attention of @Azure/aks-leads

Issue needing attention of @Azure/aks-leads

Issue needing attention of @Azure/aks-leads

Issue needing attention of @Azure/aks-leads

Issue needing attention of @Azure/aks-leads

Issue needing attention of @Azure/aks-leads

Issue needing attention of @Azure/aks-leads

Issue needing attention of @Azure/aks-leads

Issue needing attention of @Azure/aks-leads

Issue needing attention of @Azure/aks-leads

Issue needing attention of @Azure/aks-leads

Issue needing attention of @Azure/aks-leads

Issue needing attention of @Azure/aks-leads

Issue needing attention of @Azure/aks-leads

Issue needing attention of @Azure/aks-leads

Issue needing attention of @Azure/aks-leads