Kerberos broken in recent nanoserver images

[avin3sh]

Describe the bug It looks like nanoserver-ltsc2022 images released in April 2024 and later have broken Kerberos.

To Reproduce First, create a simple ASP.NET Core project AspNetKerbHello.

Create AspNetKerbHello.csproj with the following content:

<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <TargetFramework>net8.0-windows</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.AspNetCore.Authentication.Negotiate" Version="8.0.8" />
  </ItemGroup>

</Project>

Now, create Program.cs with the following content - we are enabling negotiate authentication here:

using Microsoft.AspNetCore.Authentication.Negotiate;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddAuthentication(NegotiateDefaults.AuthenticationScheme)
    .AddNegotiate();

builder.Services.AddAuthorization(options =>
{
    options.FallbackPolicy = options.DefaultPolicy;
});

var app = builder.Build();

app.UseAuthentication();
app.UseAuthorization();

app.MapGet("/hello", (HttpContext context) =>
{
    var userName = context.User.Identity?.Name ?? "Anonymous";
    var authType = context.User.Identity?.AuthenticationType ?? "None";
    return Results.Ok($"Hello, {userName}! - via {authType}");
});

app.Run();

Now build the above project, using dotnet build command.

Now, inside the build output directory where AspNetKerbHello.dll is present, create a Dockerfile with the following content. We are purposefully pulling .NET runtime 8.0.4 because it uses nanoserver image from April 2024 - i.e. mcr.microsoft.com/windows/nanoserver:ltsc2022-KB5036909 -- but this works with even the latest nanoserver image:

FROM mcr.microsoft.com/dotnet/aspnet:8.0.4-nanoserver-ltsc2022
WORKDIR /app
EXPOSE 80
ENV ASPNETCORE_URLS=http://+:80

COPY . .

USER ContainerUser
ENTRYPOINT ["dotnet", "AspNetKerbHello.dll"]

Now build this docker image, let's assume it's tagged aspnetkerbhello:v1:

docker build -t aspnetkerbhello:v1 .

Now run this image in Kubernetes as a gMSA that owns a SPN

Now, access the Kubernetes SVC URI - assuming the SPN owned by the gMSA is http/<FQDN> where FQDN is Kubernetes SVC domain name. In the below example SPN is http/my-svc.my-namespace.svc.cluster.local:

Invoke-WebRequest -UseBasicParsing -UseDefaultCredentials -AllowUnencryptedAuthentication "http://my-svc.my-namespace.svc.cluster.local/hello"

Assuming NTLM is disabled and authentication is happening over Kerberos, you will get HTTP 500 error with an exception containing the following error callstack in the Kubernetes pod logs:

fail: Microsoft.AspNetCore.Authentication.Negotiate.NegotiateHandler[5]
      An exception occurred while processing the authentication request.
      System.Net.InternalException: Exception of type 'System.Net.InternalException' was thrown. -1073741428
         at System.Net.SecurityStatusAdapterPal.GetSecurityStatusPalFromInterop(SECURITY_STATUS win32SecurityStatus, Boolean attachException)
         at System.Net.NegotiateAuthenticationPal.WindowsNegotiateAuthenticationPal.AcceptSecurityContext(SafeFreeCredentials credentialsHandle, SafeDeleteContext& securityContext, ContextFlags requestedContextFlags, ReadOnlySpan`1 incomingBlob, ChannelBinding channelBinding, Byte[]& resultBlob, Int32& resultBlobLength, ContextFlags& contextFlags)
         at System.Net.NegotiateAuthenticationPal.WindowsNegotiateAuthenticationPal.GetOutgoingBlob(ReadOnlySpan`1 incomingBlob, NegotiateAuthenticationStatusCode& statusCode)
         at System.Net.Security.NegotiateAuthentication.GetOutgoingBlob(ReadOnlySpan`1 incomingBlob, NegotiateAuthenticationStatusCode& statusCode)
         at System.Net.Security.NegotiateAuthentication.GetOutgoingBlob(String incomingBlob, NegotiateAuthenticationStatusCode& statusCode)
         at Microsoft.AspNetCore.Authentication.Negotiate.NegotiateState.GetOutgoingBlob(String incomingBlob, BlobErrorType& status, Exception& error)
         at Microsoft.AspNetCore.Authentication.Negotiate.NegotiateHandler.HandleRequestAsync()

If you made a mistake somewhere, you will instead get authenticated over NTLM and won't see the error - rather a 200 OK page with the following message:

"Hello, contoso\\username! - via NTLM"

The issue only occurs when using Kerberos, not NTLM.

Now, all of the above can be also done without involving Kubernetes with a docker container running with gMSA cred-spec file but it's bit tricky - you need to first enable Kerberos on loopback using DisableLoopbackCheck registry key, and then use [System.Net.AuthenticationManager]::CustomTargetNameDictionary to map localhost to the SPN owned by the gMSA; and even then you might endup with NTLM. So I recommend using Kubernetes for testing the above.

You do not need multiple replicas. Even just a single pod runs into the above error.

I know the issue isn't because of any change in the .NET runtime version because I have created a custom .NET runtime image with the exact same runtime version (8.0.4) but older (March 2024) nanoserver base image and I couldn't repro the issue - suggesting the issue is in the nanoserver base image.

Expected behavior

Kerberos should work with nanoserver images

Configuration:

• Edition: Windows Server 2022 with September 2024 build
• Base Image being used: nanoserver-ltsc2022
• Container engine: docker and containerd both
• Container Engine version: docker 26 / containerd 1.6.31

Additional context

Per https://github.com/dotnet/runtime/discussions/105567#discussion-6980650 the error System.Net.InternalException: Exception of type 'System.Net.InternalException' was thrown. -1073741428 translates to ERROR_TRUSTED_DOMAIN_FAILURE or The trust relationship between the primary domain and the trusted domain failed, so I suspect this is likely related to the other gMSA issue https://github.com/microsoft/Windows-Containers/issues/405