Azure
[BUG] azure function ManagedIdentity request exception during GetToken
Issue Transfer
This issue has been transferred from the Azure SDK for .NET repository, #49912.
Please be aware that @patrick-hofer is the author of the original issue and include them for any questions or replies.
Azure SDK Analysis
This is expected behavior of the credential when token acquisition fails. In this case, the cause is a failure with the local MI endpoint in the application host machine. The Azure.Identity credential only makes an HTTP request according to the MI contract and has no insight nor influence over the availability or responsiveness of the endpoint.
Investigation by the team that owns the host machine environment is needed.
Details
Describe the bug
Since 28. April some of our azure functions suddenly started to get the following exception when calling the system-assigned ManagedIdentity getToken endpoint using the ManagedIdentityCredentials class.
ManagedIdentityCredential authentication unavailable. Multiple attempts failed to obtain a token from the managed identity endpoint.
The error does not occur always, in our development environment it occurs in 1 out of 10 requests. Playing around with the retry policy (as suggested by the error message) did not fix the problem.
Azure.Identity.CredentialUnavailableException: ManagedIdentityCredential authentication unavailable. Multiple attempts failed to obtain a token from the managed identity endpoint. ---> MSAL.NetCore.4.67.2.0.MsalServiceException: ErrorCode: managed_identity_request_failed Microsoft.Identity.Client.MsalServiceException: Retry failed after 6 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80)) (An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80)) (An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80)) (An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80)) (An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80)) (An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80)) ---> System.AggregateException: Retry failed after 6 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80)) (An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80)) (An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80)) (An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80)) (An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80)) (An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80)) ---> Azure.RequestFailedException: An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80) ---> System.Net.Http.HttpRequestException: An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80) ---> System.Net.Sockets.SocketException (10013): An attempt was made to access a socket in a way forbidden by its access permissions. at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|285_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken) --- End of inner exception stack trace --- at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(QueueItem queueItem) at System.Threading.Tasks.TaskCompletionSourceWithCancellation1.WaitWithCancellationAsync(CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) at Azure.Core.Pipeline.HttpClientTransport.ProcessSyncOrAsync(HttpMessage message, Boolean async) --- End of inner exception stack trace --- at Azure.Core.Pipeline.HttpClientTransport.ProcessSyncOrAsync(HttpMessage message, Boolean async) at Azure.Core.Pipeline.HttpPipelineTransportPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline) at Azure.Core.Pipeline.RequestActivityPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) --- End of inner exception stack trace --- at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async) at Azure.Core.Pipeline.HttpPipeline.SendRequestAsync(Request request, CancellationToken cancellationToken) at Azure.Core.HttpPipelineMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) at Microsoft.Identity.Client.Http.HttpManager.ExecuteAsync(Uri endpoint, IDictionary2 headers, HttpContent body, HttpMethod method, X509Certificate2 bindingCertificate, HttpClient customHttpClient, ILoggerAdapter logger, CancellationToken cancellationToken) at Microsoft.Identity.Client.Http.HttpManager.SendRequestAsync(Uri endpoint, IDictionary2 headers, HttpContent body, HttpMethod method, ILoggerAdapter logger, Boolean doNotThrow, X509Certificate2 bindingCertificate, HttpClient customHttpClient, CancellationToken cancellationToken, Int32 retryCount) at Microsoft.Identity.Client.ManagedIdentity.AbstractManagedIdentity.AuthenticateAsync(AcquireTokenForManagedIdentityParameters parameters, CancellationToken cancellationToken) ---> (Inner Exception #1) Azure.RequestFailedException: An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80) ---> System.Net.Http.HttpRequestException: An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80) ---> System.Net.Sockets.SocketException (10013): An attempt was made to access a socket in a way forbidden by its access permissions. at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|285_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken) --- End of inner exception stack trace --- at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpC...
Multiple azure functions that are untouched since multiple months show this behaviour as well as newly created ones.
The error looks like this in azure application insights:
Expected behavior
ManagedIdentityCredential should successfully get the token and proceed with execution.
Actual behavior
The token cant be aquired leading to an exception and a failed operation.
Reproduction Steps
We use Azure.DigitalTwins.Core 1.4.0 to get data from the Azure Digital Twins service.
The DigitalTwinsClient is registered as singleton and gets an instance of ManagedIdentityCredential as TokenCredential
new DigitalTwinsClient(new Uri(url), new ManagedIdentityCredential(), ...
Later on the client.GetDigitalTwin method is called which triggers the getToken
Environment
Details regarding our DEV environment:
Azure function(s): .net 8 isolated 32bit using system assigned managed identity
OS: Windows Location: West Europe Runtime version: 4.1038.300.25164
App service plan: Pricing: Y1 Zone redundant: Disabled Consumption
I got an answer at the microsoft Q&A see azure function ManagedIdentity request exception during GetToken
TL;DR
Platform bug since end of April Happens on Windows Functions running in Consumption or Y1 plans on 32-bit Workaround: switch function to 64bit or to Linux
EDIT: switching to 64bit did not resolve the problem
The issue is still occuring regularly and the version ~4.1045 is still not available in region west europe... It regularly disrupts our processes and became very annoying in the meantime. 📉
Version 4.1045 does not yet exist, so that is not a viable workaround suggestion. Pinning to that now will effectively pin you to latest, until we move past that version eventually. In general, pinning is considered somewhat dangerous and is not recommended.
I'm unaware of a known bitness issue from the asserted timeframe, though I am asking around about that.
One thing to double-check is if you can see the IDENTITY_ENDPOINT and IDENTITY_HEADER environment variables. The easiest way to confirm these is to navigate to the app in the Azure portal, and under "Development Tools", select "Advanced Tools". Within the Kudu portal that takes you to, select "Environment" in the top banner, and try to locate these settings. Their absence indicates that the system is not aware of any identity for the app. This could be due to the identity not actually being enabled for the app or potentially some platform issue.
Could you please provide a correlation ID for a failing invocation and/or a support ticket ID?
@mattchenderson Thank you very much for the detailed infos.
I could easily locate the settings within Kudu, both IDENTITY_ENDPOINT (http://127.0.0.1:41179/msi/token/) and IDENTITY_HEADER are set and seem valid.
Environment:
- Instance id: bc487a0164505c00fe51dbb06f83d1b978109a9bca9172287dac0c9e5d887acb
- OS version: Microsoft Windows NT 10.0.20348.0
- 64 bit system: True
- 64 bit process: True
Example Invocation:
- InvocationId: f42c8431-1840-464e-a3eb-36d43136b844
- OperationId: 2c97d31439518dd9f8faa19855ff3145
- ApplicationId: 97aff22c-f374-43c3-9d72-2d49bb765bf7
- RoleInstance: 4993a783ceb8011c7d9279ab6a6e6a2569e17b64fa32c9ddee5a8c0095468261
After a while, the number of exceptions decreased. I suspect that Microsoft is slowly getting the cloud under control again and may have fixed a few bugs without notifying anyone (as I have already seen happen with some other issues).
The last function in which I still regularly encountered exceptions has now been suspiciously quiet since I upgraded to the latest Azure.Identity (1.15.0).
See: https://github.com/Azure/azure-sdk-for-net/blob/Azure.Identity_1.15.0/sdk/identity/Azure.Identity/CHANGELOG.md
For me personally, the changelog does not directly explain why this is the case, but I will continue to monitor it and hope that the problems will now be resolved.
See also my MS Q&A post about this issue: https://learn.microsoft.com/en-us/answers/questions/2275840/azure-function-managedidentity-request-exception-d
FYI @mattchenderson @Frulfump