[EKS] [request]: Control Plane Logs to S3

[mike-stewart]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request EKS currently supports control plane logging to CloudWatch Logs (#26), but it would be helpful to support S3 as an alternative log destination. This would be in line with many other AWS services that support logging both to both CloudWatch and S3.

Which service(s) is this request for? EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? Enabling S3 as a log destination would help with a few issues.

Cost: CloudWatch Logs is much more expensive than S3 for both ingestion and storage of logs, and it can easily cost in the hundreds of dollars a month to ingest the control plane logs even for a small cluster. In my experience, the cost of CloudWatch Log ingestion for many clusters exceeds the cost of EKS itself. For customers that have compliance requirements that require the collection of audit logs but do not need all the features of Cloudwatch Logs, archiving logs to S3 would be much more economical.

Flexibility: Archiving logs to S3 would allow customers that don't use Cloudwatch Logs more flexibility, such as Athena or cost-effective ingestion into another tool like AWS Elasticsearch or Splunk.

Are you currently working around this issue? Ingesting logs through CloudWatch logs, at a much higher cost.

[pparth]

@tabern is there an update here? As @mike-stewart mentioned, the cost of CloudWatch log ingestion for big clusters is prohibiting, especially when we need to have access to audit logs. We need to have the logs directly published to an S3 bucket for further processing.

[StealthyDev]

is there a workaround to this?

[aravindh-seeneevasan]

Please prioritise this. This definitely saves cost for lots of customers across the globe

[youssemn]

is there any update about this issue ?

[eightnoteight]

@tabern please prioritise this. currently our eks cluster is ingesting about 40TB per month which is a huge waste of cost for us. such cost issues are prohibiting us from moving our other big workloads from ecs/ec2 to eks.

[debu99]

Any update?

[dogzzdogzz]

I created a feature request support ticket and my TAM told me that this feature is on planning roadmap to be released in 12+ months, hopefully this feature can be prioritized because EKS logs grows quite fast with our cluster scale keeping growing and the cloudwatch is much more expensive than s3 for storing logs

[kderck]

This would be great.

[ghost]

@tabern Can we push this sooner?

[jsokol89]

not gonna lie this would be great to have!

[aravindh-seeneevasan]

We currently storing the logs to s3 via cloudwatchlog -> lambdasubscription(process logs) ->s3 -> fanout to multiple destinations via SNS

Note: Log group retention set to 5 days for reducing the cost

[dogzzdogzz]

@aravindh-seeneevasan The data ingestion cost of cloudwatch log is much more expensive than the storage, moving log from cloudwatch to s3 can not save much.

[zeppelinen]

Any known workarounds? We need audit logs but their costs are enormous.

[ClaytonOlleyNutrien]

Cloudwatch is far too expensive for these kinds of logs that need to be fed into our SIEM anyway. Please give us a direct to s3 option!

[JeffreyRaleighHall-KR]

As Einstein said: "Everything should be made as simple as possible, but no simpler". EKS may be unusable for us until it has ability to deliver Audit logs without needing Cloudwatch.

[RichardoC]

Any known workarounds? We need audit logs but their costs are enormous.

If you (@zeppelinen) only need mutations you can use https://github.com/RichardoC/kube-audit-rest

Unfortunately if you need to log read only actions, there's no other option at this time.

[etendards]

I heard this feature is not even on the EKS roadmap. We have made several requests for this feature.

[max-tremblay]

As a cost mitigation solution, AWS released the infrequent access logs option in CloudWatch last November, which aims to reduce costs by half for those who do not require any advanced features:

https://aws.amazon.com/fr/blogs/aws/new-amazon-cloudwatch-log-class-for-infrequent-access-logs-at-a-reduced-price/

[joebowbeer]

@moebaca According to the table of capabilities, Infrequent Access does not support export to S3.

IOW, those needing audit logs in S3, e.g., for 3rd party integrations, will not be able to benefit from the savings provided by Infrequent Access.

[jfarrell]

@moebaca this is incorrect, the table you reference is for Cloudwatch to push logs outbound after ingestion and that cost shown in that table is for you to pay the added push, not a comparison.

S3 ingest is $0.00

Additionally EKS does not expose the ability to set the CloudWatch Storage Class when creating the log group and CloudWatch does not allow for the storage class to be modified after creation.

EKS Team we need some help here with either S3 as a push option or exposing a way to tune the CloudWatch Storage Class or provide which CloudWatch log group to use as a configuration so it can be created with whatever settings we want.

[techmouse84]

It's sad to see that this is still an issue after so many years. My guess is that it affects AWS bottom line. Anyway, Aws Security Lake offers to get ur audit logs for you to s3 at the same price as cloudwatch IA. Just need to be aware there's lambda cost as well.