Azure Identity => ERROR in getToken() call for scopes []: Managed Identity authentication is not available.

[yashpalslathia21]

I am using using azure-identity library with version 1.12.0. I have followed all the steps to enabled workload-identity as mentioned in https://learn.microsoft.com/en-us/azure/aks/workload-identity-deploy-cluster. Following is the snippet of code I am using -

DefaultAzureCredential managedIdentityCredentialUserAssigned = new DefaultAzureCredentialBuilder() .managedIdentityClientId("bd947a20-baf1-4009-ab9a-c8aa361527a6").build();

    AccessToken accessToken = managedIdentityCredentialUserAssigned
            .getToken(new TokenRequestContext().setTenantId(tenantId)).block();
    String token = accessToken.getToken();
    logger.info("token : {}", token);

Here bd947a20-baf1-4009-ab9a-c8aa361527a6 is clientId corresponding to the managed Identity. managedIdentityCredentialUserAssigned.getToken() is throwing following error -

2024-05-09 06:07:41.760 [main] [DEBUG] com.azure.core.implementation.ReflectionUtils - Attempting to use java.lang.invoke package to handle reflection. 2024-05-09 06:07:41.763 [main] [DEBUG] com.azure.core.implementation.ReflectionUtils - Successfully used java.lang.invoke package to handle reflection. 2024-05-09 06:07:41.771 [main] [DEBUG] com.azure.identity.EnvironmentCredential - Azure Identity => Found the following environment variables: AZURE_CLIENT_ID, AZURE_TENANT_ID 2024-05-09 06:07:41.772 [main] [DEBUG] com.azure.identity.EnvironmentCredential - Azure Identity => ERROR in EnvironmentCredential: Failed to create a ClientSecretCredential or ClientCertificateCredential. Missing required environment variable either AZURE_CLIENT_SECRET or AZURE_CLIENT_CERTIFICATE_PATH 2024-05-09 06:07:41.773 [main] [DEBUG] com.azure.identity.EnvironmentCredential - Azure Identity => ERROR in EnvironmentCredential: Failed to determine an authentication scheme based on the available environment variables. Please specify AZURE_TENANT_ID and AZURE_CLIENT_SECRET to authenticate through a ClientSecretCredential; AZURE_TENANT_ID and AZURE_CLIENT_CERTIFICATE_PATH to authenticate through a ClientCertificateCredential; or AZURE_USERNAME and AZURE_PASSWORD to authenticate through a UserPasswordCredential. 2024-05-09 06:07:41.909 [main] [DEBUG] com.azure.identity.ManagedIdentityCredential - Azure Identity => Found the following environment variables: AZURE_CLIENT_ID, AZURE_TENANT_ID 2024-05-09 06:07:41.910 [main] [DEBUG] com.azure.identity.SharedTokenCacheCredential - Azure Identity => Found the following environment variables: AZURE_CLIENT_ID, AZURE_TENANT_ID

[DEBUG] com.azure.identity.ManagedIdentityCredential - Azure Identity => ERROR in getToken() call for scopes []: Managed Identity authentication is not available. 2024-05-09 06:07:42.012 [main] [INFO] com.azure.identity.ChainedTokenCredential - Azure Identity => Attempted credential EnvironmentCredential is unavailable. 2024-05-09 06:07:42.038 [main] [DEBUG] com.azure.core.implementation.util.Providers - Using com.azure.core.http.netty.NettyAsyncHttpClientProvider as the default com.azure.core.http.HttpClientProvider. 2024-05-09 06:07:42.117 [main] [WARN] com.azure.core.http.netty.implementation.Utility - The following Netty dependencies have versions that do not match the versions specified in the azure-core-http-netty pom.xml file. This may result in unexpected behavior. If your application runs without issue this message can be ignored, otherwise please update the Netty dependencies to match the versions specified in the pom.xml file. Versions found in runtime: 'io.netty:netty-codec' version: 4.1.100.Final (expected: 4.1.101.Final) 2024-05-09 06:07:43.361 [ForkJoinPool.commonPool-worker-1] [INFO] com.azure.identity.ChainedTokenCredential - Azure Identity => Attempted credential WorkloadIdentityCredential is unavailable. 2024-05-09 06:07:43.370 [ForkJoinPool.commonPool-worker-1] [DEBUG] com.azure.identity.ManagedIdentityCredential - Azure Identity => ERROR in getToken() call for scopes []: Managed Identity authentication is not available. 2024-05-09 06:07:43.370 [ForkJoinPool.commonPool-worker-1] [INFO] com.azure.identity.ChainedTokenCredential - Azure Identity => Attempted credential ManagedIdentityCredential is unavailable. 2024-05-09 06:07:43.375 [ForkJoinPool.commonPool-worker-2] [DEBUG] com.azure.identity.implementation.IdentityClient - SharedTokenCacheCredential authentication unavailable. No accounts were found in the cache.

[github-actions[bot]]

@billwert @g2vinay

[github-actions[bot]]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[yashpalslathia21]

Changed the code like this - TokenCredential managedIdentityCredential = (new ManagedIdentityCredentialBuilder()).clientId(clientId) .build(); String accessToken = ((AccessToken) managedIdentityCredential.getToken((new TokenRequestContext()) .addScopes(new String[] { "https:///.default" })).block()) .getToken();

Still getting error like this - 2024-05-09 12:21:05.499 [ForkJoinPool.commonPool-worker-1] [ERROR] com.azure.identity.ManagedIdentityCredential - Azure Identity => ERROR in getToken() call for scopes [https:///.default]: Managed Identity authentication is not available.

Can you pls confirm if this is a bug in the SDK that needs to be fixed. Is there an alternate way to fetch AAD Token for workload identity?

[yashpalslathia21]

At the moment, this is blocking me to implement workload identity.

[billwert]

Hello! Can you help me understand the scenario? Generally these credentials are used in the context of one of our service clients (such as KeyVaultClient.) Is that also failing, and you are simplifying the repro here? Can you try a scope like https://vault.azure.net or https://management.azure.com?

[yashpalslathia21]

Hi, I need to use the workload identity flow with Azure database for MySQL single server. I don't need to use the vault as it is not a requirement for us. I have followed the steps as listed in this microsoft documentation https://learn.microsoft.com/en-us/azure/aks/workload-identity-deploy-cluster . Following is the code I am using to fetch AAD token -

Map<String, String> env = System.getenv();

String tenantId = env.get("AZURE_TENANT_ID");

String clientId = env.get("AZURE_CLIENT_ID");

String tokenFile = env.get("AZURE_FEDERATED_TOKEN_FILE");

TokenCredential managedIdentityCredential = (new ManagedIdentityCredentialBuilder()).clientId(clientId)

                .build();

String accessToken = ((AccessToken) managedIdentityCredential

                .getToken((new TokenRequestContext()).addScopes(new

String[] { "https://ssp-demo-db8.mysql.database.azure.com/.default" })). block()).getToken();

But when I am running this code as K8S pod, I am getting this error -

2024-05-10 07:44:08.990 [main] [DEBUG] com.azure.identity.ManagedIdentityCredential - Azure Identity => Found the following environment variables: AZURE_CLIENT_ID, AZURE_TENANT_ID

2024-05-10 07:44:09.131 [main] [DEBUG] com.azure.core.implementation.util.Providers - Using com.azure.core.http.netty.NettyAsyncHttpClientProvider as the default com.azure.core.http.HttpClientProvider.

2024-05-10 07:44:10.636 [ForkJoinPool.commonPool-worker-1] [ERROR] com.azure.identity.ManagedIdentityCredential - Azure Identity => ERROR in getToken() call for scopes [ https://ssp-demo-db8.mysql.database.azure.com/.default]: Managed Identity authentication is not available.

CredentialUnavailableException: Managed Identity authentication is not available.}, [com.azure.identity.implementation.IdentityClient.lambda$authenticateWithManagedIdentityConfidentialClient$25(IdentityClient.java:563), reactor.core.publisher.Mono.lambda$onErrorMap$28(Mono.java:3783), reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:94), reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:241), reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:315), reactor.core.publisher.MonoCompletionStage$MonoCompletionStageSubscription.apply(MonoCompletionStage.java:119), reactor.core.publisher.MonoCompletionStage$MonoCompletionStageSubscription.apply(MonoCompletionStage.java:71), java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934), java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911), java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510), java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1773), java.base/java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1760), java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373), java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182), java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655), java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622), java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)]

Can you please help to correct or fix this issue as I have been blocked for the last 2 weeks? I have already opened an official case (Case 2405100030001925) with Microsoft for this issue and there has been no progress till now.

Regards, Yashpal

On Fri, May 24, 2024 at 4:17 AM Bill Wert @.***> wrote:

Hello! Can you help me understand the scenario? Generally these credentials are used in the context of one of our service clients (such as KeyVaultClient.) Is that also failing, and you are simplifying the repro here? Can you try a scope like https://vault.azure.net or https://management.azure.com?

— Reply to this email directly, view it on GitHub https://github.com/Azure/azure-sdk-for-java/issues/40090#issuecomment-2128156664, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATLYJAXECR7LDQSGOWGSS6DZDZWYBAVCNFSM6AAAAABHOHTNZOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRYGE2TMNRWGQ . You are receiving this because you authored the thread.Message ID: @.***>

[roumn]

I had the same issue, with debugging I figured out that the workload identity did not have the appropriate rights for the requested scope.

The token exchange API (https://login.microsoftonline.com/<tenantId>/oauth2/v2.0/token) does return the following JSON body:

{
    "error": "invalid_grant",
    "error_description": "AADSTS501051: Application '<censored app id>'(<censored app name>) is not assigned to a role for the application 'api://<censored api id>'(<censored api name>). Trace ID: <censored trace id> Correlation ID: <censored correlation id> Timestamp: <censored timestamp>",
    "error_codes": [
        501051
    ],
    "timestamp": "<censored timestamp>",
    "trace_id": "<censored trace id>",
    "correlation_id": "<censored correlation id>",
    "error_uri": "https: //login.microsoftonline.com/error?code=501051"
}

But the azure identity library (I am using 1.12.2) does not print any error message and the response body is swallowed because the status code returned is 400, and the HttpURLConnection class, which executes this call ignores the body for this response code.

[g2vinay]

I had the same issue, with debugging I figured out that the workload identity did not have the appropriate rights for the requested scope.

The token exchange API (https://login.microsoftonline.com/<tenantId>/oauth2/v2.0/token) does return the following JSON body:

{
    "error": "invalid_grant",
    "error_description": "AADSTS501051: Application '<censored app id>'(<censored app name>) is not assigned to a role for the application 'api://<censored api id>'(<censored api name>). Trace ID: <censored trace id> Correlation ID: <censored correlation id> Timestamp: <censored timestamp>",
    "error_codes": [
        501051
    ],
    "timestamp": "<censored timestamp>",
    "trace_id": "<censored trace id>",
    "correlation_id": "<censored correlation id>",
    "error_uri": "https: //login.microsoftonline.com/error?code=501051"
}

But the azure identity library (I am using 1.12.2) does not print any error message and the response body is swallowed because the status code returned is 400, and the HttpURLConnection class, which executes this call ignores the body for this response code.

Thank you for pointing this out. We will ensure, the underlying exception flows through in the exception message.

@yashpalslathia21 this is likely a config/permissions issue.

Let us know, if any assistance on API end is still needed,

[arikogan]

Hi @roumn. How did you manage to get the response body? As you say, the exception with the body doesn't bubble up and I couldn't find loggers that would allow me to get access to it. Thanks.

[billwert]

Hey @arikogan - can you try setting AZURE_HTTP_LOG_DETAIL_LEVEL=BODY_AND_HEADERS and see if that gives you what you're looking for? ref doc.

[arikogan]

Hi @billwert, thanks for the message. I tried the env var you mentioned to no luck. I also tried to configure it by code (snippet below). I'm wondering if these mechanisms apply to Azure Identity and msal4j. I would have liked to enable wire logging for the HTTP requests but there doesn't seem to be such thing for HttpURLConnection. Any idea of what else can I try to troubleshoot this? Thanks.

HttpLogOptions options = new HttpLogOptions();
options.setLogLevel(HttpLogDetailLevel.BODY_AND_HEADERS);
options.setPrettyPrintBody(true);
builder.credential(new WorkloadIdentityCredentialBuilder().build());
builder.httpLogOptions(options);

Caused by: java.io.IOException: Server returned HTTP response code: 401 for URL: https://login.microsoftonline.com/<Redacted-AZURE_TENANT_ID>/oauth2/v2.0/token
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:2021)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1610)
	at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:224)
	at com.azure.identity.implementation.IdentityClientBase.authenticateWithExchangeTokenHelper(IdentityClientBase.java:812)
	at com.azure.identity.implementation.IdentityClient.lambda$authenticateWithExchangeToken$59(IdentityClient.java:1040)
	at reactor.core.publisher.MonoCallable.call(MonoCallable.java:92)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:139)
	at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1839)
	at reactor.core.publisher.MonoCacheTime$CoordinatorSubscriber.signalCached(MonoCacheTime.java:337)
	at reactor.core.publisher.MonoCacheTime$CoordinatorSubscriber.onNext(MonoCacheTime.java:354)
	at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1839)
	at reactor.core.publisher.MonoCallable.subscribe(MonoCallable.java:62)
	at reactor.core.publisher.MonoCacheTime.subscribeOrReturn(MonoCacheTime.java:143)
	at reactor.core.publisher.Mono.subscribe(Mono.java:4476)
	at reactor.core.publisher.Mono.subscribeWith(Mono.java:4606)
	at reactor.core.publisher.Mono.toFuture(Mono.java:5011)
	at com.azure.identity.implementation.IdentityClient.lambda$getWorkloadIdentityTokenProvider$67(IdentityClient.java:1403)
	at com.microsoft.aad.msal4j.AcquireTokenByAppProviderSupplier.fetchTokenUsingAppTokenProvider(AcquireTokenByAppProviderSupplier.java:75)

[billwert]

Ah. I didn't realize you were talking about WorkloadIdentityCredential here. Unfortunately, this path doesn't use the HTTP pipeline that our normal loggers use (which is what BODY_AND_HEADERS controls.) In the short term I think you're down to some sort of proxy-inspector (fiddler, wireshark, etc.)

We did recently (last week) add some further logging to this path for exceptions, but nothing that would dump the response body directly. We'll reconsider this specific area and see if we can improve this further. Tracking that in #41876

[roumn]

Hi @roumn. How did you manage to get the response body? As you say, the exception with the body doesn't bubble up and I couldn't find loggers that would allow me to get access to it. Thanks.

Hi @arikogan, as you mentioned I could also not find any loggers for this, so I had to step through the code with the debugger to figure this out.

[g2vinay]

The status to improve logging in this area, can be tracked at https://github.com/Azure/azure-sdk-for-java/issues/41876 Closing this issue.