Duplicate response body when running on Lambda and manipulating response body stream in middleware

[madmox]

Describe the bug

I have a ASP.NET middleware which temporarily replaces the HTTP response body stream with a memory stream so that it can perform work involving seek operations on the response stream before the response is sent to the client. The code works properly when running locally using Kestrel or the integration tests server, but when running on AWS, all response bodies are duplicated (i.e. {"foo":"bar"} becomes {"foo":"bar"}{"foo":"bar"}).

Expected Behavior

The response should be a valid JSON result on all hosting environments (e.g. {"foo":"bar"}).

Current Behavior

The response is invalid when running on AWS Lambda only (duplicated JSON, e.g. {"foo":"bar"}{"foo":"bar"}).

Reproduction Steps

Simplified middleware code looks like this:

public async Task InvokeAsync(HttpContext context)
{
    Stream originalResponseBodyStream = context.Response.Body;
    using var buffer = new MemoryStream();
    context.Response.Body = buffer;

    try
    {
        await this._next(context);
        // Do work involving seek on body stream
    }
    finally
    {
        context.Response.Body.Seek(0, SeekOrigin.Begin);
        await context.Response.Body.CopyToAsync(originalResponseBodyStream);
        context.Response.Body = originalResponseBodyStream;
    }
}

Possible Solution

No response

Additional Information/Context

My ASP.NET application is using the Minimal API hosting pattern. The AWS Lambda hosting mode is set to LambdaEventSource.RestApi.

AWS .NET SDK and/or Package version used

• Amazon.Lambda.AspNetCoreServer 7.1.0
• Amazon.Lambda.AspNetCoreServer.Hosting 1.1.0

Targeted .NET Platform

.NET 6

Operating System and version

AmazonLinux

[madmox]

Sample repo to reproduce the issue.

[ashishdhingra]

Hi @madmox,

Good afternoon.

Thanks for sharing the sample code. Unfortunately, I'm unable to reproduce the issue. For reproduction,

• Cloned your code.
• Used your deployment PowerShell script to deploy the CloudFormation template.

 .\deploy.ps1 -StackName teststack-issue1185 -DeploymentBucketName testbucket-issue1880                    ❮  
D:\Lambda_Issue1185\awslambdadotnetissue1185\src\AWSLambdaTest\IndexController.cs(13,42): warning CS1998: This async method lacks 'await' operators and will run synchronously. Consider using the 'await' operator to await non-blocking API calls, or 'await Task.Run(...)' to do CPU-bound work on a background thread. [D:\Lambda_Issue1185\awslambdadotnetissue1185\src\AWSLambdaTest\AWSLambdaTest.csproj]

Build succeeded.

D:\Lambda_Issue1185\awslambdadotnetissue1185\src\AWSLambdaTest\IndexController.cs(13,42): warning CS1998: This async method lacks 'await' operators and will run synchronously. Consider using the 'await' operator to await non-blocking API calls, or 'await Task.Run(...)' to do CPU-bound work on a background thread. [D:\Lambda_Issue1185\awslambdadotnetissue1185\src\AWSLambdaTest\AWSLambdaTest.csproj]
    1 Warning(s)
    0 Error(s)

Time Elapsed 00:00:06.91
Amazon Lambda Tools for .NET Core applications (5.3.0)
Project Home: https://github.com/aws/aws-extensions-for-dotnet-cli, https://github.com/aws/aws-lambda-dotnet

Executing publish command
... invoking 'dotnet publish', working folder 'D:\Lambda_Issue1185\awslambdadotnetissue1185\src\AWSLambdaTest\bin\Release\net6.0\publish'
... dotnet publish "D:\Lambda_Issue1185\awslambdadotnetissue1185\src\AWSLambdaTest" --output "D:\Lambda_Issue1185\awslambdadotnetissue1185\src\AWSLambdaTest\bin\Release\net6.0\publish" --configuration "Release" --framework "net6.0" /p:GenerateRuntimeConfigurationFiles=true --runtime linux-x64 --self-contained false
... publish: Microsoft (R) Build Engine version 17.1.1+a02f73656 for .NET
... publish: Copyright (C) Microsoft Corporation. All rights reserved.
... publish:   Determining projects to restore...
... publish:   Restored D:\Lambda_Issue1185\awslambdadotnetissue1185\src\AWSLambdaTest\AWSLambdaTest.csproj (in 493 ms).
... publish: D:\Lambda_Issue1185\awslambdadotnetissue1185\src\AWSLambdaTest\IndexController.cs(13,42): warning CS1998: This async method lacks 'await' operators and will run synchronously. Consider using the 'await' operator to await non-blocking API calls, or 'await Task.Run(...)' to do CPU-bound work on a background thread. [D:\Lambda_Issue1185\awslambdadotnetissue1185\src\AWSLambdaTest\AWSLambdaTest.csproj]
... publish:   AWSLambdaTest -> D:\Lambda_Issue1185\awslambdadotnetissue1185\src\AWSLambdaTest\bin\Release\net6.0\linux-x64\AWSLambdaTest.dll
... publish:   AWSLambdaTest -> D:\Lambda_Issue1185\awslambdadotnetissue1185\src\AWSLambdaTest\bin\Release\net6.0\publish\
Zipping publish folder D:\Lambda_Issue1185\awslambdadotnetissue1185\src\AWSLambdaTest\bin\Release\net6.0\publish to D:\Lambda_Issue1185\awslambdadotnetissue1185\artifacts\AWSLambdaTest.zip
Creating directory D:\Lambda_Issue1185\awslambdadotnetissue1185\artifacts
... zipping: Amazon.Lambda.APIGatewayEvents.dll
... zipping: Amazon.Lambda.ApplicationLoadBalancerEvents.dll
... zipping: Amazon.Lambda.AspNetCoreServer.dll
... zipping: Amazon.Lambda.AspNetCoreServer.Hosting.dll
... zipping: Amazon.Lambda.Core.dll
... zipping: Amazon.Lambda.Logging.AspNetCore.dll
... zipping: Amazon.Lambda.RuntimeSupport.dll
... zipping: Amazon.Lambda.Serialization.SystemTextJson.dll
... zipping: appsettings.json
... zipping: AWSLambdaTest
... zipping: AWSLambdaTest.deps.json
... zipping: AWSLambdaTest.dll
... zipping: AWSLambdaTest.pdb
... zipping: AWSLambdaTest.runtimeconfig.json
Created publish archive (D:\Lambda_Issue1185\awslambdadotnetissue1185\artifacts\AWSLambdaTest.zip).
Lambda project successfully packaged: D:\Lambda_Issue1185\awslambdadotnetissue1185\artifacts\AWSLambdaTest.zip
Amazon Lambda Tools for .NET Core applications (5.3.0)
Project Home: https://github.com/aws/aws-extensions-for-dotnet-cli, https://github.com/aws/aws-lambda-dotnet

Processing CloudFormation resource ServerlessApi
Processing CloudFormation resource AspNetCoreFunction
Initiate packaging of . for resource AspNetCoreFunction
Uploading to S3. (Bucket: testbucket-issue1880 Key: AWSLambdaTest-637886578304268713.zip)
... Progress: 45%
... Progress: 91%
... Progress: 100%
Uploading to S3. (Bucket: testbucket-issue1880 Key: teststack-issue1185-cloudformation-637886578309076380.yaml)
... Progress: 100%
Found existing stack: False
CloudFormation change set created
... Waiting for change set to be reviewed
Created CloudFormation stack teststack-issue1185

Timestamp            Logical Resource Id                      Status
-------------------- ---------------------------------------- ----------------------------------------
5/20/2022 3:37 PM    teststack-issue1185                      CREATE_IN_PROGRESS
5/20/2022 3:37 PM    LambdaExecutionIamRole                   CREATE_IN_PROGRESS
5/20/2022 3:37 PM    LambdaExecutionIamRole                   CREATE_IN_PROGRESS
5/20/2022 3:37 PM    LambdaExecutionIamRole                   CREATE_COMPLETE
5/20/2022 3:37 PM    AspNetCoreFunction                       CREATE_IN_PROGRESS
5/20/2022 3:37 PM    AspNetCoreFunction                       CREATE_IN_PROGRESS
5/20/2022 3:37 PM    AspNetCoreFunction                       CREATE_COMPLETE
5/20/2022 3:37 PM    ServerlessApi                            CREATE_IN_PROGRESS
5/20/2022 3:37 PM    ServerlessApi                            CREATE_IN_PROGRESS
5/20/2022 3:37 PM    ServerlessApi                            CREATE_COMPLETE
5/20/2022 3:37 PM    AspNetCoreFunctionRootResourcePermissionProd CREATE_IN_PROGRESS
5/20/2022 3:37 PM    AspNetCoreFunctionProxyResourcePermissionProd CREATE_IN_PROGRESS
5/20/2022 3:37 PM    ServerlessApiDeploymentcfb7a37fc3        CREATE_IN_PROGRESS
5/20/2022 3:37 PM    AspNetCoreFunctionProxyResourcePermissionProd CREATE_IN_PROGRESS
5/20/2022 3:37 PM    AspNetCoreFunctionRootResourcePermissionProd CREATE_IN_PROGRESS
5/20/2022 3:37 PM    ServerlessApiDeploymentcfb7a37fc3        CREATE_IN_PROGRESS
5/20/2022 3:37 PM    ServerlessApiDeploymentcfb7a37fc3        CREATE_COMPLETE
5/20/2022 3:37 PM    ServerlessApiProdStage                   CREATE_IN_PROGRESS
5/20/2022 3:37 PM    ServerlessApiProdStage                   CREATE_IN_PROGRESS
5/20/2022 3:37 PM    ServerlessApiProdStage                   CREATE_COMPLETE
5/20/2022 3:38 PM    AspNetCoreFunctionProxyResourcePermissionProd CREATE_COMPLETE
5/20/2022 3:38 PM    AspNetCoreFunctionRootResourcePermissionProd CREATE_COMPLETE
5/20/2022 3:38 PM    teststack-issue1185                      CREATE_COMPLETE
Stack finished updating with status: CREATE_COMPLETE

• Test the changes (this works fine)

Executing dotnet --version gives output 6.0.202.

Please let me know if I'm missing any step.

Thanks, Ashish

[madmox]

Oh sorry, I made a last-minute change to my reproduction sample and it seems to be related to the buggy behavior! I updated the repo, just deployed the sample, and could reproduce the issue again.

It seems the bug has to do with the following lines:

context.Response.Body.Seek(0, SeekOrigin.Begin);
await context.Response.Body.CopyToAsync(originalResponseBodyStream);
context.Response.Body = originalResponseBodyStream;

If I replace this section with the following, it works as expected:

buffer.Seek(0, SeekOrigin.Begin);
await buffer.CopyToAsync(originalResponseBodyStream);
context.Response.Body = originalResponseBodyStream;

Although buffer and context.Response.Body are supposed to be the same object! (context.Response.Body = buffer)