Feature request: CloudWatch Infrequent Access log class support for metrics

[alfaproject]

Use case

We generate terabytes of logs per month and thus will switch all of our CloudWatch log to the new Infrequent Access log class to basically save 50% of log costs which is quite substantial: https://aws.amazon.com/blogs/aws/new-amazon-cloudwatch-log-class-for-infrequent-access-logs-at-a-reduced-price/

However, that means no support for EMF and thus the metrics middleware will stop working )':

I'm hereby suggesting to add support to send metrics directly in addition to EMF, perhaps with another adapter?

Solution/User Experience

Exactly the same APIs used in metrics middleware but instead of logging EMF to CloudWatch, just use the CloudWatch API to publish the metrics directly as an alternate adapter/solution

Alternative solutions

We are building our own custom middleware to be able to support this but eh, support here would be lovely

Acknowledgment

• [X] This feature request meets Powertools for AWS Lambda (TypeScript) Tenets
• [X] Should this be considered in other Powertools for AWS Lambda languages? i.e. Python, Java, and .NET

Future readers

Please react with 👍 and your use case to help us understand customer demand.

[boring-cyborg[bot]]

Thanks for opening your first issue here! We'll come back to you as soon as we can. In the meantime, check out the #typescript channel on our Powertools for AWS Lambda Discord: Invite link

[am29d]

Hey @alfaproject thanks for raising the PR!

This is definitely an important topic to tackle. The team is partially back from reInvent now and we have already had first discussions. We'd need a bit more time to align on all runtimes to provide consistent experience across languages and will follow up shortly on this matter.

[dreamorosi]

It's important to note that adding API calls will add latency to each invocation upon flushing and depending on how many metrics you emit it will also incur into additional pricing, which could end up decreasing or cancelling the savings derived from using the infrequent access class.

Either way, let's leave the issue open for a while and revisit in a few months based on customer feedback.

[alfaproject]

@dreamorosi that's why it's an alternate adapter and doesn't have to be the default. Whatever exists is fine, but we'll need another way to publish metrics.

I guess we'll do our own metrics middleware then but it would have been nice to use this library.

[dreamorosi]

cc @aws-powertools/lambda-python-core folks, what do you think about this feature request?

I've already expressed my opinion above but I'm copying you since it was marked also for other languages.

[rubenfonseca]

I believe this makes sense, since we already implement it in a way that allows easily to add more adapters. So we could benefit from a CloudWatch API provider in this case.

[leandrodamascena]

Sorry @dreamorosi, I completely missed this issue!

I think it makes sense for us to create a new provider with a CloudWatch API under the hood. Our current provider using EMF is very good because customer don't have network overhead, nor do they have to pay for the cost of APICall, but with these new improvements like LogClass (which doesn't support EMF), or even better error handling/visibility than we have with EMF, customers can continue to take advantage of the excellent experience provided by Powertools and use the provider with API Calls.

We should consider this provider in @aws-powertools/lambda-python-core as well.

[dreamorosi]

Revisiting the issue after a few months - so far our position remains unchanged.

We think that there could be value in adding a Metrics provider that publishes metrics using the Metrics API, but so far there hasn't been any material interest.

We'll check back in a few months to see if things have changed.