chore: AOT support for Logging and Metrics

[hjgraca]

Please provide the issue number

Issue number: #212

Summary

• Logging AOT support
• Metrics AOT support

Changes

• Logging
  • Remove Logging lambda context reflection
  • Add PowertoolsSourceGenerationContext for types discovery at compile time
• Metrics
  • Add MetricsSerializationContext for types discovery at compile time

Checklist

Please leave checklist items unchecked if they do not apply to your change.

• [x] Meets tenets criteria
• [x] I have performed a self-review of this change
• [x] Changes have been tested
• [ ] Changes are documented
• [x] PR title follows conventional commit semantics

Is this a breaking change?

RFC issue number:

Checklist:

• [ ] Migration process documented
• [ ] Implement warnings (if it can live side by side)

Acknowledgment

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.

[momo333]

Hi @hjgraca I tried version 1.6.0-alpha and I hit a couple of issues on call of the lambda. First I am getting a NullReferenceException in PowertoolsLogger::ToDictionary(), because the value of one of the properties of the object which is being converted to a dictionary is Null, and obj.GetType() call throws the exception.

The second issue that I found is that this method cannot cope with a custom POCOs, which are passed as arguments in the log event. Instead of converting them to a dictionary, a method just passed them to the serializer and it fails.

[hjgraca]

Hi @hjgraca I tried version 1.6.0-alpha and I hit a couple of issues on call of the lambda. First I am getting a NullReferenceException in PowertoolsLogger::ToDictionary(), because the value of one of the properties of the object which is being converted to a dictionary is Null, and obj.GetType() call throws the exception.

The second issue that I found is that this method cannot cope with a custom POCOs, which are passed as arguments in the log event. Instead of converting them to a dictionary, a method just passed them to the serializer and it fails.

@momo333 Thanks for testing the alpha and providing feedback. Can you share the code snippets and the error messages you are getting?

For the second issue, might be related to source generation, you must specify what classes you want to serialize, so you must include your POCOs. The code should be located at the bottom of your Lambda function, something like this: Replace with your POCOs

[JsonSerializable(typeof(YourPocoGoesHere))]
[JsonSerializable(typeof(APIGatewayProxyRequest))]
public partial class LambdaFunctionJsonSerializerContext : JsonSerializerContext
{
    // By using this partial class derived from JsonSerializerContext, we can generate reflection free JSON Serializer code at compile time
    // which can deserialize our class and properties. However, we must attribute this class to tell it what types to generate serialization code for.
    // See https://docs.microsoft.com/en-us/dotnet/standard/serialization/system-text-json-source-generation
}

[momo333]

Thank you for the fast response @hjgraca. Please find below a simple example

[assembly: LambdaGlobalProperties(GenerateMain = true)]
[assembly: LambdaSerializer(typeof(SourceGeneratorLambdaJsonSerializer<CustomJsonSerializerContext>))]
namespace SimpleLambdaWithAOT;
public class Function
{
    private readonly ILogger<Function> _logger;

    public Function(ILogger<Function> logger)
    {
        _logger = logger;
    }

    [LambdaFunction]
    [Logging(LogEvent = true, Service = "SimpleLambdaWithAOT")]
    public async Task<Response> FunctionHandlerAsync(Input request,  ILambdaContext context)
    {
        _logger.LogInformation($"Invoked lambda");

        return new Response
        {
            PropertyOne = request.PropertyOne,
            PropertyTwo = request.PropertyTwo
        };
    }
}

[JsonSerializable(typeof(Input))]
[JsonSerializable(typeof(Response))]
public partial class CustomJsonSerializerContext : JsonSerializerContext
{
}

Classes

public class Input
{
  public string PropertyOne { get; set; }

  public int PropertyTwo { get; set; }
}

public class Response
{
    public string PropertyOne { get; set; }

    public int PropertyTwo { get; set; }
}

Error message System.InvalidOperationException: 'Reflection-based serialization has been disabled for this application. Either use the source generator APIs or explicitly configure the 'JsonSerializerOptions.TypeInfoResolver' property.'

From what I saw in the source code, the JsonSerializerOptions object passed in the Serialize() method only refers to the PowertoolsSourceGenerationContext.Default, which contains the source generated serialzation logic for standard types, but does not include the custom source generated types.

[hjgraca]

Thank you for the fast response @hjgraca. Please find below a simple example

[assembly: LambdaGlobalProperties(GenerateMain = true)]
[assembly: LambdaSerializer(typeof(SourceGeneratorLambdaJsonSerializer<CustomJsonSerializerContext>))]
namespace SimpleLambdaWithAOT;
public class Function
{
    private readonly ILogger<Function> _logger;

    public Function(ILogger<Function> logger)
    {
        _logger = logger;
    }

    [LambdaFunction]
    [Logging(LogEvent = true, Service = "SimpleLambdaWithAOT")]
    public async Task<Response> FunctionHandlerAsync(Input request,  ILambdaContext context)
    {
        _logger.LogInformation($"Invoked lambda");

        return new Response
        {
            PropertyOne = request.PropertyOne,
            PropertyTwo = request.PropertyTwo
        };
    }
}

[JsonSerializable(typeof(Input))]
[JsonSerializable(typeof(Response))]
public partial class CustomJsonSerializerContext : JsonSerializerContext
{
}

Classes

public class Input
{
  public string PropertyOne { get; set; }

  public int PropertyTwo { get; set; }
}

public class Response
{
    public string PropertyOne { get; set; }

    public int PropertyTwo { get; set; }
}

Error message System.InvalidOperationException: 'Reflection-based serialization has been disabled for this application. Either use the source generator APIs or explicitly configure the 'JsonSerializerOptions.TypeInfoResolver' property.'

From what I saw in the source code, the JsonSerializerOptions object passed in the Serialize() method only refers to the PowertoolsSourceGenerationContext.Default, which contains the source generated serialzation logic for standard types, but does not include the custom source generated types.

Thanks for the example, I tried to replicate but could not reproduce the error. Some follow up questions:

• You are using LambdaAnnotations correct?
• What is the implementation of ILogger, is it Powertools or Microsoft Logging extensions?
• If you remove the [Logging(LogEvent = true, Service = "SimpleLambdaWithAOT")] does it work?

If I create a simple lambda with Powertools it works as expected, example:

public class Function
{
    private static async Task Main()
    {
        Func<string, ILambdaContext, string> handler = FunctionHandler;
        await LambdaBootstrapBuilder.Create(handler, new SourceGeneratorLambdaJsonSerializer<LambdaFunctionJsonSerializerContext>())
            .Build()
            .RunAsync();
    }


    [Logging(LogEvent = true, Service = "AOT Demo PT")]
    [Metrics(CaptureColdStart = true, Namespace = "PT Demo NS")]
    [Tracing(Namespace = "PT Demo NS", CaptureMode = TracingCaptureMode.ResponseAndError)]
    public static string FunctionHandler(string input, ILambdaContext context)
    {

        Metrics.AddMetric("metric1", 1, MetricUnit.Count);

        Metrics.AddDimension("functionVersion", "$LATEST");
        Metrics.AddMetric("Time", 100.5, MetricUnit.Milliseconds, MetricResolution.Standard);

        var customKeys = new Dictionary<string, string>
        {
            {"test1", "value1"},
            {"test2", "value2"}
        };
        Logger.AppendKeys(customKeys);

        Logger.AppendKey("New Key", "AOT is awesome!");

        Logger.LogInformation($"Hello from Powertools! Function name: " + context.FunctionName);

        Tracing.AddAnnotation("Annotation1", "My Anottation");

        Logger.RemoveKeys("test1", "test2");


        return input.ToUpper();
    }
}

[JsonSerializable(typeof(string))]
[JsonSerializable(typeof(APIGatewayProxyRequest))]
public partial class LambdaFunctionJsonSerializerContext : JsonSerializerContext
{
}

[momo333]

Thank you for the response.

You are using LambdaAnnotations correct?

As far as I am aware I am using it correctly, it is working properly without the AOT part.

What is the implementation of ILogger, is it Powertools or Microsoft Logging extensions?

The implementation is Powertools.Logging.

If you remove the [Logging(LogEvent = true, Service = "SimpleLambdaWithAOT")] does it work?

Even without the attribute, a simple log information call is failing.

Have you tried to pass POCO instead of string in the Function Handlers input parameter (like in my example above )?

public async Task<Response> FunctionHandlerAsync(Input request, ILambdaContext context) then you should hit the problem.

[hjgraca]

Have you tried to pass POCO instead of string in the Function Handlers input parameter (like in my example above )?

public async Task<Response> FunctionHandlerAsync(Input request, ILambdaContext context) then you should hit the problem.

The bellow code is straight from the Hello World example of Sam cli, I have added Powertools logger and decorated the handler and added a Log information in the body and works

The function receives as input a APIGatewayHttpApiV2ProxyRequest instead of string.

In your scenario Instead of using _logger from the injected ILogger, can you use the static method Logger.LogInformation($"Invoked lambda");

public class Function
{
    private static readonly HttpClient client = new HttpClient();

    /// <summary>
    /// The main entry point for the Lambda function. The main function is called once during the Lambda init phase. It
    /// initializes the .NET Lambda runtime client passing in the function handler to invoke for each Lambda event and
    /// the JSON serializer to use for converting Lambda JSON format to the .NET types. 
    /// </summary>
    private static async Task Main()
    {
        Func<APIGatewayHttpApiV2ProxyRequest, ILambdaContext, Task<APIGatewayHttpApiV2ProxyResponse>> handler = FunctionHandler;
        await LambdaBootstrapBuilder.Create(handler, new SourceGeneratorLambdaJsonSerializer<LambdaFunctionJsonSerializerContext>())
            .Build()
            .RunAsync();
    }

    private static async Task<string> GetCallingIP()
    {
        client.DefaultRequestHeaders.Accept.Clear();
        client.DefaultRequestHeaders.Add("User-Agent", "AWS Lambda .Net Client");

        var msg = await client.GetStringAsync("http://checkip.amazonaws.com/").ConfigureAwait(continueOnCapturedContext: false);

        return msg.Replace("\n", "");
    }

    [Logging(LogEvent = true, Service = "AOT Demo PT")]
    public static async Task<APIGatewayHttpApiV2ProxyResponse> FunctionHandler(APIGatewayHttpApiV2ProxyRequest apigProxyEvent, ILambdaContext context)
    {

        var location = await GetCallingIP();
        var body = new Dictionary<string, string>
        {
            { "message", "hello world" },
            { "location", location }
        };

        Logger.LogInformation($"Hello from: " + location);

        return new APIGatewayHttpApiV2ProxyResponse
        {
            Body = JsonSerializer.Serialize(body, typeof(Dictionary<string, string>), LambdaFunctionJsonSerializerContext.Default),
            StatusCode = 200,
            Headers = new Dictionary<string, string> { { "Content-Type", "application/json" } }
        };
    }
}

[JsonSerializable(typeof(APIGatewayHttpApiV2ProxyRequest))]
[JsonSerializable(typeof(APIGatewayHttpApiV2ProxyResponse))]
[JsonSerializable(typeof(Dictionary<string, string>))]
public partial class LambdaFunctionJsonSerializerContext : JsonSerializerContext
{
    // By using this partial class derived from JsonSerializerContext, we can generate reflection free JSON Serializer code at compile time
    // which can deserialize our class and properties. However, we must attribute this class to tell it what types to generate serialization code for.
    // See https://docs.microsoft.com/en-us/dotnet/standard/serialization/system-text-json-source-generation
}

[momo333]

Hi @hjgraca, I tried your example above and it is working without an issue on my side as well. I also played out several scenarios and maybe the problem is not the POCO, but the generic type (we are passing indeed input with generic types)

Please find another example below, I hope this one is reproducible on your side as well

public class Function
{
    private static async Task Main()
    {
        Func<Generic<Input>, ILambdaContext, Response> handler = FunctionHandler;
        await LambdaBootstrapBuilder.Create(handler, new SourceGeneratorLambdaJsonSerializer<LambdaFunctionJsonSerializerContext>())
            .Build()
            .RunAsync();
    }

    [Logging(LogEvent = true, Service = "AOT Demo")]
    public static Response FunctionHandler(Generic<Input> input, ILambdaContext context)
    {
        return new Response() { };
    }
}

[JsonSerializable(typeof(Generic<Input>))]
[JsonSerializable(typeof(Input))]
[JsonSerializable(typeof(Response))]
public partial class LambdaFunctionJsonSerializerContext : JsonSerializerContext
{
}

public class Generic<T> 
{
    public T PropertyOne { get; set; }
}

public class Input
{
    public string PropertyOne { get; set; }
    public string PropertyTwo { get; set; }
}

 public class Response
 {
     public string PropertyOne { get; set; }
 }

[hjgraca]

Hi @hjgraca, I tried your example above and it is working without an issue on my side as well. I also played out several scenarios and maybe the problem is not the POCO, but the generic type (we are passing indeed input with generic types)

Please find another example below, I hope this one is reproducible on your side as well

public class Function
{
    private static async Task Main()
    {
        Func<Generic<Input>, ILambdaContext, Response> handler = FunctionHandler;
        await LambdaBootstrapBuilder.Create(handler, new SourceGeneratorLambdaJsonSerializer<LambdaFunctionJsonSerializerContext>())
            .Build()
            .RunAsync();
    }

    [Logging(LogEvent = true, Service = "AOT Demo")]
    public static Response FunctionHandler(Generic<Input> input, ILambdaContext context)
    {
        return new Response() { };
    }
}

[JsonSerializable(typeof(Generic<Input>))]
[JsonSerializable(typeof(Input))]
[JsonSerializable(typeof(Response))]
public partial class LambdaFunctionJsonSerializerContext : JsonSerializerContext
{
}

public class Generic<T> 
{
    public T PropertyOne { get; set; }
}

public class Input
{
    public string PropertyOne { get; set; }
    public string PropertyTwo { get; set; }
}

 public class Response
 {
     public string PropertyOne { get; set; }
 }

Hi @momo333 I am running the exact same code you shared and is working. What is the payload you are sending? how are you deploying and what tools you are using for development? If you want we can go on a call and try to troubleshoot what is going on.

[momo333]

The payload that I am sending is:

{ "propertyOne": { 
"propertyOne": "test"
  } 
}

The tools for development are standard - Visual Studio 2022 with AWS Toolkit for Visual Studio, with dotnet-lambda-test-tool-8.0 for testing locally. A call is a great option, I would be very grateful if you could help .

[hjgraca]

dotnet-lambda-test-tool-8.0

Sent you an email. Have you deployed the Lambda function to AWS or are you running on the test tool only? Also make sure you are using version 1.6.0-alpha of Powertools.

[momo333]

Hi again @hjgraca, yes I was referring Powertools version 1.6.0-alpha. I have tried it locally and I deployed it as well (both ways the error was the same).

There was an internal exception thrown in the background, that the code was handling gracefully. After some debugging and adding the options (and some custom trimming options for our project)

<JsonSerializerIsReflectionEnabledByDefault>false</JsonSerializerIsReflectionEnabledByDefault>
<TrimMode>partial</TrimMode>
<PublishTrimmed>true</PublishTrimmed>

I think that resolved the error and I started receiving NullReferenceException which is probably related to the first issue I mentioned, the one with the property value being null. You can find a pull request for that one here https://github.com/hjgraca/powertools-lambda-dotnet/pull/1. I hope it makes sense

[sonarqubecloud[bot]]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarCloud

[momo333]

thank you for your time @hjgraca, it is much appreciated

[JDurstberger]

Hello.

I am running into issues when using the updated logging lib. I see the following runtime error:

Unhandled Exception: System.NotSupportedException: 
'AWS.Lambda.Powertools.Common.UniversalWrapperAttribute.WrapAsync[System.Threading.Tasks.VoidTaskResult](System.Func`2[System.Object[],System.Threading.Tasks.Task`1[System.Threading.Tasks.VoidTaskResult]],System.Object[],AWS.Lambda.Powertools.Common.AspectEventArgs)' is missing native code. MethodInfo.MakeGenericMethod() is not compatible with AOT compilation. 
Inspect and fix AOT related warnings that were generated when the app was published. 
For more information see https://aka.ms/nativeaot-compatibility

This seems to happen immediately at start up. I am setting the logger on an .NET minimal API using

builder.Logging.ClearProviders();
builder.Logging.AddProvider(new LoggerProvider());

Where LogProvider is

using AWS.Lambda.Powertools.Logging;

public class LoggerProvider : ILoggerProvider
{
  public ILogger CreateLogger(string categoryName)
  {
    return Logger.Create(categoryName);
  }

  public void Dispose()
  {
    GC.SuppressFinalize(this);
  }
}

Any idea what could be the root cause?

[hjgraca]

Hey @JDurstberger what version are you using? Is this for an AOT enabled application? The AOT version is still alpha, and I haven't tested yet with minimal apis AOT.

[JDurstberger]

I was using 1.6.0-alpha with an AOT enabled application. It looks to me like the serialization fails on messages logged by the minimal api itself.

[hjgraca]

released on #606 #619 #647