Workload Identity: Failed to acquire a token

[lukaseisenring]

Describe the bug The token of Workload Identity cannot get acquired. Logs are reporting an incomplete environment variable configuration and in documentation no more possible values are specified.

The configuration is identical to the Testing section in the PR https://github.com/Azure/application-gateway-kubernetes-ingress/pull/1498 of @akshaysngupta

To Reproduce Steps to reproduce the behavior:

Install AGIC using Helm (Version 1.7.1) with the following parameters:

appgw.applicationGatewayID: x armAuth.type: workloadIdentity armAuth.identityClientID: x kubernetes.securityContext.runAsUser: 1000 rbac.enabled: true verbosityLevel: 1

Ingress Controller details

• Output of kubectl describe pod <ingress controller> . The pod name can be obtained by running helm list. `Name: application-gateway-kubernetes-ingress-ingress-azure-54d9cnqll6 Namespace: application-gateway-ingress-controller Priority: 0 Service Account: application-gateway-kubernetes-ingress-sa-ingress-azure Node: x Start Time: Tue, 27 Jun 2023 07:43:46 +0200 Labels: app=ingress-azure azure.workload.identity/use=true pod-template-hash=54d9cf6f47 release=application-gateway-kubernetes-ingress Annotations: checksum/config: fe8217e775b14e18c53cd5d54b2bc7719deda1da25a2f18345e9abaf101d1181 prometheus.io/port: 8123 prometheus.io/scrape: true Status: Running IP: x IPs: IP: x Controlled By: ReplicaSet/application-gateway-kubernetes-ingress-ingress-azure-54d9cf6f47 Containers: ingress-azure: Container ID: containerd://6bde7e11a42e5c995fabd9f55ba9843ce7a2c51fdae45d5fa5e36ef8eacdeff2 Image: mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.7.1 Image ID: mcr.microsoft.com/azure-application-gateway/kubernetes-ingress@sha256:91a6648b78c65f3b6858441589daabd72146d9a53e896c0e6abf501e870f9d9b Port: Host Port: State: Running Started: Tue, 27 Jun 2023 07:43:47 +0200 Ready: False Restart Count: 0 Liveness: http-get http://:8123/health/alive delay=15s timeout=1s period=20s #success=1 #failure=3 Readiness: http-get http://:8123/health/ready delay=5s timeout=1s period=10s #success=1 #failure=3 Environment Variables from: application-gateway-kubernetes-ingress-cm-ingress-azure ConfigMap Optional: false Environment: AZURE_CLOUD_PROVIDER_LOCATION: /etc/appgw/azure.json AGIC_POD_NAME: application-gateway-kubernetes-ingress-ingress-azure-54d9cnqll6 (v1:metadata.name) AGIC_POD_NAMESPACE: application-gateway-ingress-controller (v1:metadata.namespace) AZURE_CLIENT_ID: x AZURE_TENANT_ID: x AZURE_FEDERATED_TOKEN_FILE: /var/run/secrets/azure/tokens/azure-identity-token AZURE_AUTHORITY_HOST: https://login.microsoftonline.com/ Mounts: /etc/appgw/ from azure (ro) /var/run/secrets/azure/tokens from azure-identity-token (ro) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fbjhm (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: azure: Type: HostPath (bare host directory volume) Path: /etc/kubernetes/ HostPathType: Directory kube-api-access-fbjhm: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true azure-identity-token: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3600 QoS Class: BestEffort Node-Selectors: x Tolerations: :NoSchedule op=Exists node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message

  ---

  Normal Scheduled 56m default-scheduler Successfully assigned application-gateway-ingress-controller/application-gateway-kubernetes-ingress-ingress-azure-54d9cnqll6 to x Normal Pulling 56m kubelet Pulling image "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.7.1" Normal Pulled 56m kubelet Successfully pulled image "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.7.1" in 261.149933ms (261.155633ms including waiting) Normal Created 56m kubelet Created container ingress-azure Normal Started 56m kubelet Started container ingress-azure Warning Unhealthy 67s (x374 over 55m) kubelet Readiness probe failed: Get "http://x:8123/health/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) `

• Output of kubectl logs <ingress controller>. I0627 05:43:47.609870 1 utils.go:114] Using verbosity level 1 from environment variable APPGW_VERBOSITY_LEVEL I0627 05:43:47.616327 1 main.go:81] Unable to load cloud provider config '/etc/appgw/azure.json'. Error: Reading Az Context file "/etc/appgw/azure.json" failed: open /etc/appgw/azure.json: permission denied I0627 05:43:48.648152 1 supported_apiversion.go:70] server version is: 1.25.6 I0627 05:43:48.659580 1 environment.go:294] KUBERNETES_WATCHNAMESPACE is not set. Watching all available namespaces. I0627 05:43:48.659607 1 main.go:118] Using User Agent Suffix='application-gateway-kubernetes-ingress-ingress-azure-54d9cnqll6' when communicating with ARM I0627 05:43:48.659688 1 auth.go:58] Creating authorizer using Default Azure Credentials I0627 05:43:48.660007 1 httpserver.go:57] Starting API Server on :8123 E0627 05:44:48.660931 1 authorizer.go:46] Error getting Azure token: DefaultAzureCredential: failed to acquire a token. Attempted credentials: EnvironmentCredential: incomplete environment variable configuration. Only AZURE_TENANT_ID and AZURE_CLIENT_ID are set WorkloadIdentityCredential: unable to resolve an endpoint: server response error: context deadline exceeded `

Azure Support Request ID: 2306270050000735