Memory Stream of response body is empty while running on Lambda using middleware.

[vinodkumarkv]

Describe the bug

I have a ASP.NET middleware which copies the HTTP response body stream with a memory stream so that it can perform work involving sends logs to third party using response stream before the response is sent to the client. The code works properly when running locally using Kestrel or the integration tests server, but when running on AWS, all response body of the copied memory stream is empty, but the actual context response body of response contains the expected data.

Expected Behavior

The copied response memory stream should have the same context body response on all hosting environments

Current Behavior

The memory stream is empty when running on AWS Lambda only

Reproduction Steps

Simplified middleware code using Minimal API of AWS looks like this:

Program.cs

WebApplicationBuilder builder = WebApplication.CreateBuilder(args);
ConfigureServices(builder);

WebApplication app = builder.Build();
ConfigurePipeline(app);


app.Run();

static void ConfigureServices(WebApplicationBuilder builder)
{
    builder.Services.AddControllers();
    builder.Services.AddAWSLambdaHosting(LambdaEventSource.RestApi);
}

static void ConfigurePipeline(WebApplication app)
{
    app.UseMiddleware<ResponseBodyStreamManipulatorMiddleware>();
    app.UseRouting();
    app.MapControllers();
}

ResponseBodyStreamManipulatorMiddleware.cs

public class ResponseBodyStreamManipulatorMiddleware
   {
       private readonly RequestDelegate _next;

       public ResponseBodyStreamManipulatorMiddleware(RequestDelegate next)
       {
           this._next = next;
       }

       public async Task InvokeAsync(HttpContext context)
       {
           context.Request.EnableBuffering();
           Stream originalBodyStream = context.Response.Body;
           using var responseBody = new MemoryStream();
           context.Response.Body = responseBody;

           await _next(context);

           context.Response.Body.Seek(0, SeekOrigin.Begin);
           using var reader = new StreamReader(responseBody);
           var resBody = await reader.ReadToEndAsync();

           Console.WriteLine("Response Data" + resBody); //resBody is empty while running in AWS Lambda
           Console.WriteLine("responseBody " + resBody.Length); //resBody length is zero while running in AWS Lambda

           context.Response.Body.Seek(0, SeekOrigin.Begin);
           await responseBody.CopyToAsync(originalBodyStream);

       }
   }

IndexController.cs

[Route("")]
    [ApiController]
    public class IndexController : ControllerBase
    {
        [HttpGet]
        [Route("")]
        public ActionResult<SampleResponse> Get()
        {
            var result = new SampleResponse() { responseValue ="Test" };
            return Ok(result);
        }
    }

Possible Solution

No response

Additional Information/Context

My ASP.NET application is using the Minimal API hosting pattern. The AWS Lambda hosting mode is set to LambdaEventSource.RestApi.

Running application in Lambda:

Cloud Watch Logs memory stream of responsebody is empty :

we observe the using var responseBody = new MemoryStream(); is loosing it value even though it's pointing to same object refrerence context.Response.Body = responseBody;

Note: The reason we require the responseBody stream is because we are using it for writing audit logs (internally we are calling third party by sending the resonseBody stream and right now copied responseBody stream going empty)

AWS .NET SDK and/or Package version used

• Amazon.Lambda.AspNetCoreServer 8.0.0 • Amazon.Lambda.AspNetCoreServer.Hosting 1.5.1

Targeted .NET Platform

.NET 6

Operating System and version

AmazonLinux

[ashishdhingra]

Reproducible.

• Created new serverless.AspNetCoreMinimalAPI project using command dotnet new serverless.AspNetCoreMinimalAPI.
• Made changes as user suggested, creating extra needed classes.
• Debugged locally using Visual Studio Kestrel web server, hitting the API controller endpoint. Noted that the response data is logged.
• Deployed to AWS Lambda environment using command dotnet deploy-serverless specifying the prompted values.
• Navigated to the deployed REST API endpoint.
• Examined the CloudWatch logs to confirm the issue that the response data is logged as empty.

Unsure if this is related to another issue https://github.com/aws/aws-lambda-dotnet/issues/1185 where duplicated response is logged instead.

[philmagr]

I also have this issue, and I have narrowed down a workaround:

Root cause is this:

            using var responseBody = new MemoryStream();
            context.Response.Body = responseBody;

Does not fully update the context.Response.Body - when calling the getter the previous body is returned.

Replacing it with this:

            using var responseBody = new MemoryStream();
            var originalBodyFeature = (IHttpResponseBodyFeature)context.Features[typeof(IHttpResponseBodyFeature)];
            context.Features[typeof(IHttpResponseBodyFeature)] = new StreamResponseBodyFeature(responseBody, originalBodyFeature);

Will cause context.Response.Body to return the newly created stream.

What is the difference? Setter of context.Response.Body internally calls Set<TFeature>(TFeature instance) on the IFeatureCollection used for the request in order to update the response stream. Amazon.Lambda.AspNetCoreServer.Internal.InvokeFeatures.Set<TFeature>(TFeature instance) function does not increment _containerRevision. Amazon.Lambda.AspNetCoreServer.Internal.InvokeFeatures[Type key] does increment _containerRevision.

Examples - Replace these lines in the ResponseBodyStreamManipulatorMiddleware.cs in original issue with one of the following:

           Stream originalBodyStream = context.Response.Body;
           using var responseBody = new MemoryStream();
           context.Response.Body = responseBody;

This works, logs (except for the type of context.Features) match between local/Lambda:

            using var responseBody = new MemoryStream();
            var originalBodyStream = context.Response.Body;
            Console.WriteLine(context.Features.GetType().AssemblyQualifiedName);
            Console.WriteLine(context.Response.Body.GetType().AssemblyQualifiedName);
            Console.WriteLine(context.Features.Get<IHttpResponseBodyFeature>().Stream.GetType().AssemblyQualifiedName);
            Console.WriteLine(context.Features.Revision);
            var originalBodyFeature = (IHttpResponseBodyFeature)context.Features[typeof(IHttpResponseBodyFeature)];
            context.Features[typeof(IHttpResponseBodyFeature)] = new StreamResponseBodyFeature(responseBody, originalBodyFeature);
            Console.WriteLine(context.Features.GetType().AssemblyQualifiedName);
            Console.WriteLine(context.Response.Body.GetType().AssemblyQualifiedName);
            Console.WriteLine(context.Features.Get<IHttpResponseBodyFeature>().Stream.GetType().AssemblyQualifiedName);
            Console.WriteLine(context.Features.Revision); 

This does not work, logs do not match between local/Lambda:

            using var responseBody = new MemoryStream();
            var originalBodyStream = context.Response.Body;
            Console.WriteLine(context.Features.GetType().AssemblyQualifiedName);
            Console.WriteLine(context.Response.Body.GetType().AssemblyQualifiedName);
            Console.WriteLine(context.Features.Get<IHttpResponseBodyFeature>().Stream.GetType().AssemblyQualifiedName);
            Console.WriteLine(context.Features.Revision);
            context.Response.Body = responseBody;
            Console.WriteLine(context.Features.GetType().AssemblyQualifiedName);
            Console.WriteLine(context.Response.Body.GetType().AssemblyQualifiedName);
            Console.WriteLine(context.Features.Get<IHttpResponseBodyFeature>().Stream.GetType().AssemblyQualifiedName);
            Console.WriteLine(context.Features.Revision); 

In Lambda, Revision is not incremented, the IHttpResponseBodyFeature is actually updated, but it is never used because context.Response.Body does not reflect the updated value.