Asp.Net Core WebApi - AWS ECS Cluster Authentication failure

[jcrawfor74]

trafficstars

Is there an existing issue for this?

• [X] I have searched the existing issues

Describe the bug

This is a bit of a long shot, as it is quite a complicated issue to reproduce. I can supply more detail as required.

The issue is that we are having an authentication failure when our api is deployed on an AWS ECS cluster

We have a .net core 6.0 web api that we deploy to AWS as both a Lambda accessed via Api Gateway and as an ECS cluster.

This site has a custom authentication and authorization.

The main relevant code for this is, a ServiceCollection extension that adds our custom authorzation and policies

public static void AddWebApiAuthorisation(this IServiceCollection services)
{
    services.AddAuthorization(options =>
    {
        options.DefaultPolicy = new AuthorizationPolicyBuilder().RequireAuthenticatedUser().AddAuthenticationSchemes("custom").Build();

 options.AddPolicy(...

When hosting on AWS ECS Cluster, you have the following setup:

• Route53 URL routed to a Application Load Balancer (ALB)
• ALB rule to direct specific URL traffic to a TargetGroup
• Target Group has the connection to the ECS Cluster Tasks (Task is a running instance of the docker image)
  • The target group has a health check that internally calls the running task to confirm that it is up and running.
  • We configure a base static route /welcome that is used to call for the health check.
• The code is deployed to the ECS custer as a docker image that is built and published to the ECR, elastic container repository.

The health check is just some static html that is configured as a custom route in the startup as follows:

var welcome = GetWelcome(env.IsProduction() , basePath);
app.UseEndpoints(endpoints =>
{
    endpoints.MapControllers();
    endpoints.MapGet("/welcome", async context =>
    {
        await context.Response.WriteAsync(welcome);
    });
});

Basic the GetWelcome generates some basic html.

So the problem is as follows when accessing the /welcome url

1. In local debug - no issues
2. In local docker - no issues
3. deployed to AWS Lambda - no issues
4. deployed to AWS ECS Cluster - Authentication failure

Expected Behavior

the /welcome URL on the base of the website should render.

Steps To Reproduce

Deployed the webapi to ECS cluster, the target group health check calls /welcome and fails logging:

[Information] Microsoft.AspNetCore.Authorization.DefaultAuthorizationService: Authorization failed. These requirements were not met:	svc-webapi-users
DenyAnonymousAuthorizationRequirement: Requires an authenticated user.

? Not sure what you need to investigate..

Here is some of the WebApi Custom setup code Authentication..

var privateKey = (parameterStoreInstance.GetSecureStringAsync(ParameterStoreKeys.ApiSigningKey).Result);
var key = Encoding.ASCII.GetBytes(privateKey);


 services.AddAuthentication(options =>
 {
     options.DefaultAuthenticateScheme = JwtBearerDefaults.AuthenticationScheme;
     options.DefaultChallengeScheme = JwtBearerDefaults.AuthenticationScheme;
 })
.AddJwtBearer("custom", x =>
 {
     x.RequireHttpsMetadata = false;
     x.SaveToken = true;
     x.TokenValidationParameters = new TokenValidationParameters
     {
         ValidateIssuerSigningKey = true,
         IssuerSigningKey = new SymmetricSecurityKey(key),
         ValidateIssuer = false,
         ValidateAudience = false,
         ValidateLifetime = true,
         LifetimeValidator = (before, expires, token, param) => expires > DateTime.UtcNow
     };
 })

Setup for Web Api Versioning

public static void AddWebApiVersioning(this IServiceCollection services)
{
    services
        .AddApiVersioning(options =>
        {
            options.ReportApiVersions = true;
            options.AssumeDefaultVersionWhenUnspecified = true;
            options.DefaultApiVersion = new ApiVersion(1, 0);
        })
        .AddApiExplorer(options =>
        {
            // add the versioned api explorer, which also adds IApiVersionDescriptionProvider service
            // note: the specified format code will format the version as "'v'major[.minor][-status]"
            options.GroupNameFormat = "'v'VVV";

            // note: this option is only necessary when versioning by url segment. the SubstitutionFormat
            // can also be used to control the format of the API version in route templates
            options.SubstituteApiVersionInUrl = true;
        });
}

Startup code for swagger in startup.cs

 app.UseSwagger();
 app.UseSwaggerUI(
     options =>
     {
         // options.RoutePrefix = "docs";
         // build a swagger endpoint for each discovered API version
         foreach (var description in provider.ApiVersionDescriptions)
         {
             options.SwaggerEndpoint($"{basePath}/swagger/{description.GroupName}/swagger.json", description.GroupName.ToUpperInvariant());
         }
     });

packages running the webapi

<PackageReference Include="Microsoft.AspNet.WebApi.Client" Version="5.2.9" />
<PackageReference Include="Microsoft.AspNetCore.Mvc.NewtonsoftJson" Version="6.0.20" />
<!--<PackageReference Include="Microsoft.AspNetCore.Mvc.Versioning" Version="5.1.0" />-->
<!--<PackageReference Include="Microsoft.AspNetCore.Mvc.Versioning.ApiExplorer" Version="5.1.0" />-->
<PackageReference Include="Asp.Versioning.Mvc.ApiExplorer" Version="6.4.0" />
<PackageReference Include="Swashbuckle.AspNetCore" Version="6.5.0" />

Exceptions (if any)

[Information] Microsoft.AspNetCore.Authorization.DefaultAuthorizationService: Authorization failed. These requirements were not met:	svc-webapi-users
DenyAnonymousAuthorizationRequirement: Requires an authenticated user.

.NET Version

.Net Core 6.0

Anything else?

So the issue is very odd. The code works locally in debug, on a local docker image, and deployed to AWS Lambda, but fails when deployed as the same docker image on an ECS cluster.

Note: it is 100% the same code deployed for the same branch to both Lambda and ECS cluster and yet one works and one doesn't?

The /welcome URL is an anonymous endpoint with no authentication but it seems to be being treated as a having authorization.

I rolled back to:

Microsoft.AspNetCore.Mvc.Versioning.ApiExplorer

With no other code changes (apart from making it compile due to the library changes) deployed and it now works, returning 200 - OK on the /welcome Url

I think the /welcome URL is not the root issue I think the entire site would return a 401, but given that the 401 is causing the health check to fail the task cannot reach a stable state to be able to test it further.

I suspect you will not be able to diagnose this, but I thought I would ask in case there is something that jumps out at you as a possible root cause of the problem?

Happy to test a beta package if you think you can identify a potential fix.