containers-roadmap
containers-roadmap copied to clipboard
aws
[EKS] [Audit log]: Allow to custom audit-log policy
Tell us about your request Currently, we could only enable audit log with the default policy. Is it possible to ship audit-log-policy to config map for customization?
Which service(s) is this request for? EKS
Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? Currently, EKS use audit-log with default policy (log all request via kube-api), it causes a huge log to process on our system (most of these logs come from trusted local service). Could we custom audit policy to reduce unnecessary logs?
This is kind of important, since the audit log costs can compound quickly in cloud watch.
Do we have any plan to allow EKS logging policy customisation for EKS users ?
Currently, some audit logs do spit JWT tokens. At work (Nubank), we use centralized logging, and generally like and see value in making them available to everyone (audit logs, for example, do find casual audiences, in the form of "oh, weird that the HPA is that way, I guess if I can find who changed it maybe I can get context as to why"). The fixed policy is blocking us from doing that (which frankly, seems like a gap for us, which used to be possible running our own distro), while a custom policy would do so (as we could omit them via the policy).
I'm betting a lot of people come to this Github issue after seeing their Cloudwatch costs when enabling audit logging.
I'm echoing everyone else, please let us customize audit policy.
I'm betting a lot of people come to this Github issue after seeing their Cloudwatch costs when enabling audit logging.
I'm one of these people. Does anyone have tips & tricks to partially solve this issue ? On a simple dev cluster with only audit logging I end up with a 40$ CloudWatch bill. And I don't even want to use CloudWatch, I have my own logging system in the cluster !
I'll suggest to write few lines about why custom audit police or even log delivery to S3 is important and if lack of this feature is impacting your adoption. It will help EKS team to prioritize the request.
By when we can expect custom audit log policy in EKS ? cloud watch billing is about to touch sky.
One of the services I support recently had a misbehaving application in staging that was beating on the K8s API (list pods and nodes) and was burning $800/day on Cloudwatch alone for several days until it tripped a billing alert. Suffice it to say we immediately disabled control plane audit logging until the application was fixed. Then we had some 'splaining to do.
This should have been entirely avoidable. I don't care about auditing read-only calls, and would have filtered those out had I had the choice.
I agree with @jtackaberry. Ability to custom audit-log policy can bring us additonal cost savings. +1
I also would like to see the ability to custom the audit log policy to bring my team an additional cost savings. +1
Completely agree with the above comments, but this could be potentially provided in addition to, or as an alternative to this one. It aims to solve the same problem, even if it provides less flexibility.
https://github.com/aws/containers-roadmap/issues/1141
You can pass a file with the policy to kube-apiserver using the --audit-policy-file flag. If the flag is omitted, no events are logged. Note that the rules field must be provided in the audit policy file. A policy with no (0) rules is treated as illegal.
https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/
It's extremely disappointing that this isn't yet supported. I'm beginning to suspect the issue on AWS's end is to do with GuardDuty requiring (specific?) audit logs from the control plane to perform EKS detections..?
How important is fully customizable audit log here, vs if EKS introduced a new minimal audit log type, for example that excluded get and list requests from the logs?
@mikestef9 in my humble opinion, let the users have option and make an informed decision. Get and List requests do have significant role to play especially in security arena. Although, I know the implications of a fully bloated logs, but again, let use make an informed decision.
How important is fully customizable audit log here, vs if EKS introduced a new minimal audit log type, for example that excluded get and list requests from the logs?
Sorry to be blunt, but after 4 years, anything will do, the sooner the better, minimal audit type will save thousands of $ in cloudwatch logging, which is the main driver for this ask
It's concerning how long has it been with no answer, audit logs are mandatory to be compliant with some standards (such as PCI), so some companies can't simply turn it off, other than that since my company uses a external SIEM tool we'll have to work around filtering the logs from cloudwatch before shipping them, otherwise those logs by themselves will burn all our daily quota in just a few hours
I believe AWS recently announced that these EKS logs are at least "vended" in Cloudwatch now so they should be considerably less expensive.
Well, less expensive is not a solution. Either AWS make it completely free or empower customers to export logs wherever they want.
@mikestef9 That would help, but I would like Secret read/list audited too, and as others suggest, they have their own opinions :)
It could be beneficial to get that feature, Costs are extraordinary when every log is collected. A customization that allows you to choose which logs specifically to collect would solve that.
From Kubernetes docs, it seems like it's customizable. https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/
It's not reasonable for this issue to last for 4 years. Audit logs, which is a native feature of K8S control plane, should be fully customizable according to customers' needs. Hundreds of upvotes above shouldn't be meaning much for AWS. In the meantime, making EKS cloudwatch logs (only audit ones) free might cause customer satisfaction at least.
Verifying that the audit policy file is configured appropriately is now one of the CIS Benchmark recommendations.
Where are the contents of the actual EKS --audit-policy-file?
There is a yaml snippet in the docs source (below), but how to verify that it is actually used?
https://github.com/aws/aws-eks-best-practices/blob/master/content/security/docs/detective.md
@joebowbeer I found it in EKS Best Practices guide here. https://aws.github.io/aws-eks-best-practices/security/docs/detective/#auditing-and-logging
Sorry I'm confused, does that mean this issue is fixed by setting that policy?
@EmrhT replied:
I found it in EKS Best Practices guide here https://aws.github.io/aws-eks-best-practices/security/docs/detective/#auditing-and-logging
Thanks. I updated my question.