Intermittent ConfigurationLoadCredential exception with QUIC (when using Google GTS library): QUIC_CREDENTIAL_FLAGS flags, X509Certificate

[DanDiplo]

What version of gRPC and what language are you using?

We're using the Google.Cloud.Talent.V4 2.6.0 NuGet package for NET 8.0

This references Grpc.Net.Api 2.6.0

What operating system (Linux, Windows,...) and version?

Windows Server 2022 Datacenter Azure Edition VM Microsoft Windows NT 10.0.20348.0

What runtime / compiler are you using (e.g. .NET Core SDK version dotnet --info)

Version: 8.0.6 Architecture: x64 Commit: 3b8b000a0e

What did you do?

We are using the Google Talent Solutions .NET library 2.6.0 in an ASP.NET Core site (NET8.0) based on Umbraco 13.4.0 hosted within IIS.

We use the library to make calls to retrieve and update jobs from GTS.

What did you expect to see?

We expected the calls to always work.

What did you see instead?

Whilst 99.99% of calls work, we are getting seemingly random intermittent GRPC errors that seem related to loading credentials and the QUIC protocol.

The stack trace is like this:

Grpc.Core.RpcException: Status(StatusCode="Unavailable", Detail="Error starting gRPC call. HttpRequestException: An internal error has occurred. ConfigurationLoadCredential failed: Unknown (0x80090331) (jobs.googleapis.com:443) QuicException: An internal error has occurred. ConfigurationLoadCredential failed: Unknown (0x80090331)", DebugException="System.Net.Http.HttpRequestException: An internal error has occurred. ConfigurationLoadCredential failed: Unknown (0x80090331) (jobs.googleapis.com:443)")
 ---> System.Net.Http.HttpRequestException: An internal error has occurred. ConfigurationLoadCredential failed: Unknown (0x80090331) (jobs.googleapis.com:443)
 ---> System.Net.Quic.QuicException: An internal error has occurred. ConfigurationLoadCredential failed: Unknown (0x80090331)
   at System.Net.Quic.ThrowHelper.ThrowMsQuicException(Int32 status, String message)
   at System.Net.Quic.MsQuicConfiguration.Create(QuicConnectionOptions options, QUIC_CREDENTIAL_FLAGS flags, X509Certificate certificate, ReadOnlyCollection`1 intermediates, List`1 alpnProtocols, CipherSuitesPolicy cipherSuitesPolicy, EncryptionPolicy encryptionPolicy)
   at System.Net.Quic.MsQuicConfiguration.Create(QuicClientConnectionOptions options)
   at System.Net.Quic.QuicConnection.FinishConnectAsync(QuicClientConnectionOptions options, CancellationToken cancellationToken)
   at System.Net.Quic.QuicConnection.<ConnectAsync>g__StartConnectAsync|2_0(QuicClientConnectionOptions options, CancellationToken cancellationToken)
   at System.Net.Quic.QuicConnection.<ConnectAsync>g__StartConnectAsync|2_0(QuicClientConnectionOptions options, CancellationToken cancellationToken)
   at System.Net.Http.ConnectHelper.ConnectQuicAsync(HttpRequestMessage request, DnsEndPoint endPoint, TimeSpan idleTimeout, SslClientAuthenticationOptions clientAuthenticationOptions, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.ConnectQuicAsync(HttpRequestMessage request, DnsEndPoint endPoint, TimeSpan idleTimeout, SslClientAuthenticationOptions clientAuthenticationOptions, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttp3ConnectionAsync(HttpRequestMessage request, HttpAuthority authority, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.TrySendUsingHttp3Async(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at Grpc.Net.Client.Balancer.Internal.BalancerHttpHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Grpc.Net.Client.Internal.GrpcCall`2.RunCall(HttpRequestMessage request, Nullable`1 timeout)
   --- End of inner exception stack trace ---
   at Google.Api.Gax.Grpc.ApiCallLoggingExtensions.<>c__DisplayClass0_0`2.<<WithLogging>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Google.Api.Gax.Grpc.ApiCallRetryExtensions.<>c__DisplayClass0_0`2.<<WithRetry>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Ph.Gts.Api.Services.GtsSearchService.SearchJobsAsync(GtsSearchCriteria criteria, CallSettings callSettings, String culture) in E:\actions-runner2\_work\xxxxxxxxx\GtsSearchService.cs:line 67

In the Windows Event log around the same time we see this:

A fatal error occurred while creating a TLS client credential. The internal error state is 10013.
The SSPI client process is w3wp (PID: 13796).

The server has TLS1.3 enabled.

Anything else we should know about your project / environment?

The error is intermittent and doesn't follow any perceivable pattern. It only appears to happen on our Production server; I've never seen it when running locally in IIS Express direct from Visual Studio 2022. When it does happen, it usually happens just once in a period. From our recent logs these are the times it occurred:

Jun 28, 2024 1:12:13 AM
Jun 28, 2024 1:01:38 AM
Jun 28, 2024 12:41:11 AM
Jun 28, 2024 12:31:07 AM
Jun 28, 2024 12:20:55 AM

I can't correlate this with any particular action or event.

[ManickaP]

This looks like TLS error https://learn.microsoft.com/en-us/windows/win32/secauthn/schannel-error-codes-for-tls-and-ssl-alerts SEC_E_ALGORITHM_MISMATCH

How are the SslOptions configured? Can you share the code? Are all H/3 requests failing or just for a specific server or randomly?

BTW, the time pattern corresponds to AltSvc blocklist timeout of 10 mins: https://github.com/dotnet/runtime/blob/d9a66070026c9133fade4e61a60d00aec170bfc4/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/ConnectionPool/HttpConnectionPool.Http3.cs#L24

We get Alt-Svc, we try, we fail, we block for 10 mins, rinse and repeat.

[DanDiplo]

Hi. Thank you for responding.

We're using the Google Talent Solutions (GTS) .NET library NuGet package. so aren't calling the GRPC library directly. But the GTS library includes Google.Api.Gax.Grpc (2.6.0) which in turn uses Grpc.Net.Client (2.60.0)

https://www.nuget.org/packages/Google.Cloud.Talent.V4/2.6.0

https://developers.google.com/api-client-library/dotnet/apis/jobs/v4

So we create a GTS JobServiceClient which is the service that queries GTS and it is this that uses GRPC under-the-hood. We create the client like this in a singleton .NET class registered through DI:

       private JobServiceClient? jobServiceClient;

        public async Task<JobServiceClient> GetJobServiceClientAsync()
        {
            if (this.jobServiceClient == null)
            {
                var builder = new JobServiceClientBuilder()
                {
                    CredentialsPath = GetCredentialsPath(),
                    Logger = logger
                };

                this.jobServiceClient = await builder.BuildAsync();
            }

            return jobServiceClient;
        }

So we just use the defaults that Google sets. Unfortunately I can't see any documentation for how to configure this. The JobServiceClientBuilder does have some options for things like GrpcAdapter but I'm honestly not sure what this could be set as.

https://cloud.google.com/dotnet/docs/reference/Google.Cloud.Talent.V4/latest/Google.Cloud.Talent.V4.JobService.JobServiceClient

In terms of failures, most requests are not failing. The vast majority are working fine, it just seems to sporadically fail, though the 10 min interval you mention does seem to correlate roughly with the errors in the log:

Interestingly we use the GTS client library in a couple more ASP.NET core websites, but these are .NET 6, and they don't have this issue. The issue is only with this site that is using .NET 8. However, this site is on a different Windows VM, so that might be the factor.

The server is hosted behind Azure Front Door and uses the WAF, if that might be a factor?

According to SSL Labs test TLS1.3 and TLS1.2 are enabled:

I'm not really a server guy, but if there's any server config info I can get you that might help I can pass it on to our team. Thank you for any help or advice.

[rzikm]

Interestingly we use the GTS client library in a couple more ASP.NET core websites, but these are .NET 6, and they don't have this issue. The issue is only with this site that is using .NET 8. However, this site is on a different Windows VM, so that might be the factor.

The failure is related to HTTP/3 support, which is enabled by default since .NET 8.

I think this issue is not going to be gRPC specific. We should move it to dotnet/runtime.

I see CredentialsPath = GetCredentialsPath(), in the code, is a client certificate used for the authentication? I remember talk about client certificates not being supported in TLS 1.3 (or at least for QUIC) on Windows Server 2022 and we have these kinds of tests disabled on that platform (although I tried to run them now to check if the error is the same and all tests seem to pass without errors so I guess it is supported now?).

As a workaround, you can disable HTTP/3 using either of

• System.Net.SocketsHttpHandler.Http3Support AppContext switch (from code)
• DOTNET_SYSTEM_NET_HTTP_SOCKETSHTTPHANDLER_HTTP3SUPPORT environment variable (set to 0/false)

Looking at the source code of both .NET and MsQuic, it seems that the error comes from AcquireCredentialsHandle, this happens before even sending anything to the server, my guess is that there is something environmental which breaks QUIC support.

If my hypothesis is right, it should not matter which server you are trying to connect to, can you try something as simple as

using HttpClient client = new();
Console.WriteLine(await client.SendAsync(new HttpRequestMessage(HttpMethod.Get, "https://www.microsoft.com")
{
    Version = HttpVersion.Version30,
    VersionPolicy = HttpVersionPolicy.RequestVersionExact
}));

On the target machine and see if the issue reproduces?

To investiate further, we would need following data

Data collection instructions

TLS configuration from registry

reg export HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\Schannel schannel_settings.reg /y
reg export HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Cryptography\Configuration\Local\SSL\00010002 schannel_ciphersuites.reg /y
reg export HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Cryptography\Configuration\Local\SSL\00010003 schannel_signatures.reg /y
reg export HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Cryptography\Configuration\Local\Default\00010002 schannel_default.reg /y
reg export HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Cryptography\Configuration\SSL schannel_grouppolicy.reg /y
reg export HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa\FipsAlgorithmPolicy system_fips.reg /y
reg export "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations\RDP-Tcp" system_rdp.reg /y
reg export "HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows NT\Terminal Services" rdp_group_policy.reg /y

Enabled TLS ciphers

From powershell

(Get-TlsCipherSuite).Name > schannel_ciphersuites.txt
Get-TlsEccCurve > schannel_curves.txt

Schannel logs

logman start schannel_trace -p "{37D2C3CD-C5D4-4587-8531-4696C44244C8}" 0x7fffffff -o schannel.etl -ets -ln schannel
logman start ncryptsslp_trace -p "{A74EFE00-14BE-4ef9-9DA9-1484D5473304}” 0x7fffffff -o ncryptsslp.etl -ets -ln ncryptsslp

run repro, try to keep it as short as possible, one failure is enough.

logman stop schannel_trace -ets
logman stop ncryptsslp_trace -ets

Are you able to collect them them and share them with me? You can use my work address radekzikmund (at) microsoft.com if you don't want to post them publicly here.

[DanDiplo]

Hi. Thank you once again for your help.

I created a .NET8 console app with your HttpClient code and got this error when running on the server in question:

Unhandled exception. System.Net.Http.HttpRequestException: An internal error has occurred. ConfigurationLoadCredential failed: Unknown (0x80090331) (www.microsoft.com:443)
 ---> System.Net.Quic.QuicException: An internal error has occurred. ConfigurationLoadCredential failed: Unknown (0x80090331)
   at System.Net.Quic.ThrowHelper.ThrowMsQuicException(Int32 status, String message)
   at System.Net.Quic.MsQuicConfiguration.Create(QuicConnectionOptions options, QUIC_CREDENTIAL_FLAGS flags, X509Certificate certificate, ReadOnlyCollection`1 intermediates, List`1 alpnProtocols, CipherSuitesPolicy cipherSuitesPolicy, EncryptionPolicy encryptionPolicy)
   at System.Net.Quic.MsQuicConfiguration.Create(QuicClientConnectionOptions options)
   at System.Net.Quic.QuicConnection.FinishConnectAsync(QuicClientConnectionOptions options, CancellationToken cancellationToken)
   at System.Net.Quic.QuicConnection.<ConnectAsync>g__StartConnectAsync|2_0(QuicClientConnectionOptions options, CancellationToken cancellationToken)
   at System.Net.Quic.QuicConnection.<ConnectAsync>g__StartConnectAsync|2_0(QuicClientConnectionOptions options, CancellationToken cancellationToken)
   at System.Net.Http.ConnectHelper.ConnectQuicAsync(HttpRequestMessage request, DnsEndPoint endPoint, TimeSpan idleTimeout, SslClientAuthenticationOptions clientAuthenticationOptions, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.ConnectQuicAsync(HttpRequestMessage request, DnsEndPoint endPoint, TimeSpan idleTimeout, SslClientAuthenticationOptions clientAuthenticationOptions, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttp3ConnectionAsync(HttpRequestMessage request, HttpAuthority authority, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.TrySendUsingHttp3Async(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Program.<Main>$(String[] args) in D:\Websites\Projects\ConsoleApps\HttpTest\HttpTest\Program.cs:line 4
   at Program.<Main>(String[] args)

So this does seem to be very similar to the error, so good spot.

I will email you with the registry and trace settings shortly. Thank you.

[rzikm]

Hi, the culprit is in the enabled TLS cipher suites list. Compared to my machine, your machine has disabled the following:

TLS_AES_256_GCM_SHA384 TLS_AES_128_GCM_SHA256

These are TLS cipher suites necessary for TLS 1.3. Not sure how the server was able to use TLS 1.3 before, when I disable these on my machine, I hit the same issue as you report, and when I try to force TLS 1.3 from a client, the connection fails to establish.

You can enable these ciphersuites via administrator powershell prompt

Enable-TlsCipherSuite TLS_AES_256_GCM_SHA384
Enable-TlsCipherSuite TLS_AES_128_GCM_SHA256

Let me know if you have any questions.

[DanDiplo]

I'll pass this on to our server admin and let you know the outcome. Thanks!

[JamesNK]

@rzikm Is there a way to make the error message clearer in the client so people can self-diagnose this issue?

[rzikm]

I filed an issue in .NET runtime (it is linked above) to improve diagnostics, it should be easy to detect, but I first need to take a bit more time to confirm if the error code is unique to this situation.

[DanDiplo]

@rzikm Our server admin has now enabled the TLS ciphers you mentioned and I am happy to report that this has resolved the error. Thank you for all your help and also @ManickaP, as I would never have been able to figure this out on my own. I'm happy for you to close this if you wish.