Intermittent ConfigurationLoadCredential exception with QUIC (when using Google GTS library): QUIC_CREDENTIAL_FLAGS flags, X509Certificate

[DanDiplo]

What version of gRPC and what language are you using?

We're using the Google.Cloud.Talent.V4 2.6.0 NuGet package for NET 8.0

This references Grpc.Net.Api 2.6.0

What operating system (Linux, Windows,...) and version?

Windows Server 2022 Datacenter Azure Edition VM Microsoft Windows NT 10.0.20348.0

What runtime / compiler are you using (e.g. .NET Core SDK version dotnet --info)

Version: 8.0.6 Architecture: x64 Commit: 3b8b000a0e

What did you do?

We are using the Google Talent Solutions .NET library 2.6.0 in an ASP.NET Core site (NET8.0) based on Umbraco 13.4.0 hosted within IIS.

We use the library to make calls to retrieve and update jobs from GTS.

What did you expect to see?

We expected the calls to always work.

What did you see instead?

Whilst 99.99% of calls work, we are getting seemingly random intermittent GRPC errors that seem related to loading credentials and the QUIC protocol.

The stack trace is like this:

Grpc.Core.RpcException: Status(StatusCode="Unavailable", Detail="Error starting gRPC call. HttpRequestException: An internal error has occurred. ConfigurationLoadCredential failed: Unknown (0x80090331) (jobs.googleapis.com:443) QuicException: An internal error has occurred. ConfigurationLoadCredential failed: Unknown (0x80090331)", DebugException="System.Net.Http.HttpRequestException: An internal error has occurred. ConfigurationLoadCredential failed: Unknown (0x80090331) (jobs.googleapis.com:443)")
 ---> System.Net.Http.HttpRequestException: An internal error has occurred. ConfigurationLoadCredential failed: Unknown (0x80090331) (jobs.googleapis.com:443)
 ---> System.Net.Quic.QuicException: An internal error has occurred. ConfigurationLoadCredential failed: Unknown (0x80090331)
   at System.Net.Quic.ThrowHelper.ThrowMsQuicException(Int32 status, String message)
   at System.Net.Quic.MsQuicConfiguration.Create(QuicConnectionOptions options, QUIC_CREDENTIAL_FLAGS flags, X509Certificate certificate, ReadOnlyCollection`1 intermediates, List`1 alpnProtocols, CipherSuitesPolicy cipherSuitesPolicy, EncryptionPolicy encryptionPolicy)
   at System.Net.Quic.MsQuicConfiguration.Create(QuicClientConnectionOptions options)
   at System.Net.Quic.QuicConnection.FinishConnectAsync(QuicClientConnectionOptions options, CancellationToken cancellationToken)
   at System.Net.Quic.QuicConnection.<ConnectAsync>g__StartConnectAsync|2_0(QuicClientConnectionOptions options, CancellationToken cancellationToken)
   at System.Net.Quic.QuicConnection.<ConnectAsync>g__StartConnectAsync|2_0(QuicClientConnectionOptions options, CancellationToken cancellationToken)
   at System.Net.Http.ConnectHelper.ConnectQuicAsync(HttpRequestMessage request, DnsEndPoint endPoint, TimeSpan idleTimeout, SslClientAuthenticationOptions clientAuthenticationOptions, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.ConnectQuicAsync(HttpRequestMessage request, DnsEndPoint endPoint, TimeSpan idleTimeout, SslClientAuthenticationOptions clientAuthenticationOptions, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttp3ConnectionAsync(HttpRequestMessage request, HttpAuthority authority, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.TrySendUsingHttp3Async(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at Grpc.Net.Client.Balancer.Internal.BalancerHttpHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Grpc.Net.Client.Internal.GrpcCall`2.RunCall(HttpRequestMessage request, Nullable`1 timeout)
   --- End of inner exception stack trace ---
   at Google.Api.Gax.Grpc.ApiCallLoggingExtensions.<>c__DisplayClass0_0`2.<<WithLogging>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Google.Api.Gax.Grpc.ApiCallRetryExtensions.<>c__DisplayClass0_0`2.<<WithRetry>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Ph.Gts.Api.Services.GtsSearchService.SearchJobsAsync(GtsSearchCriteria criteria, CallSettings callSettings, String culture) in E:\actions-runner2\_work\xxxxxxxxx\GtsSearchService.cs:line 67

In the Windows Event log around the same time we see this:

A fatal error occurred while creating a TLS client credential. The internal error state is 10013.
The SSPI client process is w3wp (PID: 13796).

The server has TLS1.3 enabled.

Anything else we should know about your project / environment?

The error is intermittent and doesn't follow any perceivable pattern. It only appears to happen on our Production server; I've never seen it when running locally in IIS Express direct from Visual Studio 2022. When it does happen, it usually happens just once in a period. From our recent logs these are the times it occurred:

Jun 28, 2024 1:12:13 AM
Jun 28, 2024 1:01:38 AM
Jun 28, 2024 12:41:11 AM
Jun 28, 2024 12:31:07 AM
Jun 28, 2024 12:20:55 AM

I can't correlate this with any particular action or event.