Issue with unhandled exception message been multilined

[svoychik]

Describe the bug

As a result of unhandled exceptions, logs are written into Cloudwatch in multiline format. https://github.com/aws/aws-lambda-dotnet/pull/1165 - previous attempt to fix the issue didn't help so this probably has to be double-checked

For us, this issue causes quite a lot of garbage logs, which distracts attention from really important things

Expected Behavior

The exception message is stored as a single CloudWatch row. That's how it worked before on .NET 3.1

Current Behavior

Currently, exception logs are multilined, see screenshot

Reproduction Steps

You need to use lambda with .NET 6 AWS-supported runtime. Then make that lambda fail - Throw any exception from lambda ExecuteAsync method and open CloudWatch logs to see the multilined exception logs

Possible Solution

No response

Additional Information/Context

No response

AWS .NET SDK and/or Package version used

Amazon.Lambda.Core 2.1.0 Amazon.Lambda.Serialization.Json 2.0.0 Amazon.Lambda.APIGatewayEvents 2.4.1 Amazon.Lambda.Logging.AspNetCore 3.1.0 AWSSDK.CloudWatch 3.7.4.25 AWSSDK.DynamoDBv2 3.7.3.50 AWSSDK.StepFunctions 3.7.0.174

Targeted .NET Platform

.NET 6

Operating System and version

.NET 6 runtime

[normj]

Hi @svoychik

I pinged the Lambda team on the status of the deployment and they have been doing a lot of internal changes to how their runtime deployments. The side effect is the change made to the .NET runtime client hasn't been deployed yet to the managed runtime. Sorry for the delay.

[svoychik]

Hi @normj Thanks for the response. Is there a place where I can track the latest updates to the .NET runtime in terms of the version of packages to not spend a lot of time trying to understand whether the Lambda team released the fix or not?

Also, considering the speed of delivery (more than 3 months), it looks like a big reason to think about whether we really want to use custom runtime or the one provided by AWS 🤔.
I'd like to know the Lambda team's vision on the processes. Are such long delays still going to happen?

[DillonN]

Seeing the same issue on .NET 6 still. We did manage to find a workaround, but it's not that pretty. Basically implement a quick custom ILogger implementation that logs to Console.Out, but replaces all \n characters with \r. CloudWatch seems to pick up on the carriage return as an internal line break and formats the messages as expected.

E: Adding some context on the workaround since there was interest:

This really only applies if you're using the same Amazon.Lambda.Logging.AspNetCore mentioned in OP. It's a lightweight library from this repo that implements a Microsoft.Extensions.Logging provider using ILambdaLogger. Since ILambdaLogger stopped formatting multi-line messages properly with .NET 6, so too did this library.

The workaround is to swap out the logging library with one that logs to std out, e.g. Microsoft.Extensions.Logging.Console. Then modify this new library to replace all \n with \r in every log message. If you use the same ILogger[<T>] interface everywhere, this is a drop-in replacement.

But unfortunately you can't actually use Microsoft.Extensions.Logging.Console, since the relevant code is internal and sealed. What you can do is implement a custom logging provider that makes the \n->\r swap. This ends up being two classes to implement ILogger and ILoggerProvider. Inspiration on e.g. formatting can be taken from the Amazon.Lambda.Logging.AspNetCore implementation. Keep in mind logging can be a sensitive component, make sure everyone understands the risks of rolling a custom implementation. Also, you'll probably want the \n->\r thing turned off in local debugging.

This workaround came up years ago for a custom .NET Core 3.1 runtime, but after migrating to 3.1 official it wasn't necessary. After dusting it off to try on this problem in .NET 6, it works!

[BenjaRav]

@ashishdhingra I see this is marked as queued now. Is it possible to have an estimate for how long tasks like this usually resides in a queued state, before being started? I'm contemplating continuing to try to implement the fix Dillon suggested here, but i feel it will be wasted if a internal fix from AWS is around the corner.

[BenjaRav]

Any updates on this?

[ashishdhingra]

The changes are pushed, closing this issue. Feel free to open a new one in case of any issue.

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

[SaaimaMulla]

May I know which release is this fix part of. I have updated all the NuGet packages for AWS but hasn't fix this for me. @ashishdhingra

[ashishdhingra]

May I know which release is this fix part of. I have updated all the NuGet packages for AWS but hasn't fix this for me. @ashishdhingra

@SaaimaMulla Per https://github.com/aws/aws-lambda-dotnet/issues/1265#issuecomment-1204134370, it's the Lambda runtime fix/change. Please advise:

• How you are deploying your Lambda. Are you deploying it as a container.
• What region are you using for deployment?

Thanks, Ashish

[SaaimaMulla]

May I know which release is this fix part of. I have updated all the NuGet packages for AWS but hasn't fix this for me. @ashishdhingra

@SaaimaMulla Per #1265 (comment), it's the Lambda runtime fix/change. Please advise:

• How you are deploying your Lambda. Are you deploying it as a container.
• What region are you using for deployment?

Thanks, Ashish

Yes it is deployed in containers for region Europe-Ireland

[niels-foreflight]

I don't know who to tag, since the last tag to @ashishdhingra didn't get replied. So i'm tagging @normj as well.

We are also still experiencing this issue with unhandled exceptions being printed as multiline. Our Lambdas s running in us-east-1. As docker containers using the newest base image from public.ecr.aws/lambda/dotnet:6, but the issue still persist.

[normj]

@niels-foreflight It is a limitation with the running functions as a container image that exceptions get written out as multiline. Basically in containers Lambda is capturing stdout from the container to send CloudWatch Logs and with that there is no way to know the end of a log message other then making the assumption on newlines. The fix we did before only affected when deploying as a zip file.

[pjmvp]

On ECS one can configure the AWS log driver with awslogs-multiline-pattern.

Any way something similar can be achieved with Lambda running functions as a container image?

[pjmvp]

@normj , any plans on making it possible to have these logs w/ exceptions in a single log entry, similar to what one does with ECS? Or are there any workarounds you'd recommend? Thank you.