[BUG] Persistence check failed exception on Windows

[hoarfrostm]

trafficstars

Library name and version

azure.identity 1.10.4

Describe the bug

An error occured while we use .Net Azure.Identiy named persistent token cache options.

Code:

TokenCredential tokenCredential = new ClientCertificateCredential(
             TenantId,
             ClientId,
             authCert,
             new ClientCertificateCredentialOptions
             {
                SendCertificateChain = true,
                TokenCachePersistenceOptions = new TokenCachePersistenceOptions
                {
                   Name = $"{Id}-{ClientId}"
                }
             }
         );

TableServiceClient tableServiceClient = new TableServiceClient(
                new Uri(tableUrl),
                tokenCredential,
                DefaultTableClientOptions
            );
   
TableClient tableClient = tableServiceClient.GetTableClient(tableName);

Pageable<EntityNew> records = tableClient.Query<EntityNew>();
List<X> ips = new List<X>();
foreach (EntityNew record in records)
{
    if (!string.IsNullOrWhiteSpace(record.X))
    {
    ...
        }
    }
}          

Exception:

Azure.Identity.AuthenticationFailedException: ClientCertificateCredential authentication failed: Persistence check failed. Data was written but it could not be read. Possible cause: on Linux, LibSecret is installed but D-Bus isn't running because it cannot be started over SSH.

See the troubleshooting guide for more information.
https://aka.ms/azsdk/net/identity/clientcertificatecredential/troubleshoot
---> Microsoft.Identity.Client.Extensions.Msal.MsalCachePersistenceException: Persistence check failed. Data was written but it could not be read. Possible cause: on Linux, LibSecret is installed but D-Bus isn't running because it cannot be started over SSH.

   at Microsoft.Identity.Client.Extensions.Msal.Storage.VerifyPersistence()

   at Azure.Identity.MsalCacheHelperWrapper.VerifyPersistence()

   at Azure.Identity.TokenCache.<GetCacheHelperAsync>d__26.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at Azure.Identity.TokenCache.<GetCacheHelperAsync>d__26.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Azure.Identity.TokenCache.<RegisterCache>d__21.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Azure.Identity.MsalClientBase`1.<GetClientAsync>d__26.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Azure.Identity.MsalConfidentialClient.<AcquireTokenForClientCoreAsync>d__21.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Azure.Identity.MsalConfidentialClient.<AcquireTokenForClientAsync>d__20.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Azure.Identity.ClientCertificateCredential.GetToken(TokenRequestContext requestContext, CancellationToken cancellationToken)

   --- End of inner exception stack trace ---

   at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex, String additionalMessage, Boolean isCredentialUnavailable)

   at Azure.Identity.ClientCertificateCredential.GetToken(TokenRequestContext requestContext, CancellationToken cancellationToken)

   at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.<GetHeaderValueFromCredentialAsync>d__9.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.<GetHeaderValueAsync>d__6.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.<GetHeaderValueAsync>d__6.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AuthenticateAndAuthorizeRequest(HttpMessage message, TokenRequestContext context)

   at Azure.Data.Tables.TableBearerTokenChallengeAuthorizationPolicy.<AuthorizeRequestInternal>d__6.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Azure.Data.Tables.TableBearerTokenChallengeAuthorizationPolicy.AuthorizeRequest(HttpMessage message)

   at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.<ProcessAsync>d__11.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)

   at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory`1 pipeline)

   at Azure.Core.Pipeline.RedirectPolicy.<ProcessAsync>d__7.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Azure.Core.Pipeline.RedirectPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)

   at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory`1 pipeline)

   at Azure.Core.Pipeline.RetryPolicy.<ProcessAsync>d__5.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at Azure.Core.Pipeline.RetryPolicy.<ProcessAsync>d__5.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

   at Azure.Core.Pipeline.RetryPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)

   at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory`1 pipeline)

   at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)

   at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory`1 pipeline)

   at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)

   at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory`1 pipeline)

   at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)

   at Azure.Core.Pipeline.HttpPipeline.Send(HttpMessage message, CancellationToken cancellationToken)

   at Azure.Data.Tables.TableRestClient.QueryEntities(String table, Nullable`1 timeout, String nextPartitionKey, String nextRowKey, QueryOptions queryOptions, CancellationToken cancellationToken)

   at Azure.Data.Tables.TableClient.<>c__DisplayClass56_0`1.<Query>b__0(Nullable`1 pageSizeHint)

   at Azure.Core.PageableHelpers.FuncPageable`1.<AsPages>d__4.MoveNext()

   at Azure.Pageable`1.<GetEnumerator>d__8.MoveNext()

Expected behavior

No exception

Actual behavior

threw Persistence check failed exception

Reproduction Steps

Randomly happened multiple times in multiple services. Most frequently after a reboot of VM. But Not necessarily.

Environment

Server OS Version 6.3 (20348) - Windows Server 2022 Datacenter

.Net Framework 4.7.2

"nuget": [ { "id": "azure.core", "version": "1.36.0", "platform": "windows" }, { "id": "azure.identity", "version": "1.10.4", "platform": "windows" }

[jsquire]

@hoarfrostm: It is not recommended that you set TokenCachePersistenceOptions as a general practice. Those options are intended for unusual scenarios, such as when you are writing a CLI that requires durable state persistence. Setting those options causes the normal memory-based cache of the credential to be persisted to/from disk for every operation, which is a large performance hit and can cause issues in some environments when the local OS key store is not available.

To mitigate, we advise not setting the TokenCachePersistenceOptions.

[github-actions[bot]]

Hi @hoarfrostm. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text "/unresolve" to remove the "issue-addressed" label and continue the conversation.

[rayluo]

It is the encryption that is not always successful. There can be a middle ground of still persisting token cache on disk, but without encryption.

Azure Identity's underlying library MSAL .Net supports turning off encryption: "The cross-platform token cache allows you to store unencrypted tokens in an ACL-restricted plain-text file. This is useful in cases where encryption at rest fails, which ocassionally happens due to environmnent-related reasons."

But I'm not sure whether Azure Identity exposes that behavior. Azure Identity's doc seemingly hints that it would always enable token encryption when running on Windows.

[github-actions[bot]]

Hi @hoarfrostm, since you haven’t asked that we /unresolve the issue, we’ll close this out. If you believe further discussion is needed, please add a comment /unresolve to reopen the issue.