containers-roadmap
containers-roadmap copied to clipboard
aws
[EKS] [request]: Custom EKS service IAM role w/o AWS managed policies
Today the EKS service IAM role specified for cluster creation MUST include the AWS managed policies AmazonEKSServicePolicy and AmazonEKSClusterPolicy. The CreateCluster API returns an error if either is missing from the specified role.
In a regulated setting we require control over all the IAM roles used and so cannot use AWS managed policies. We require the ability to specify an IAM role without the AWS managed policies during cluster creation (of course taking on the responsibility to ensure that the role we specify has the required permissions).
@ewbankkit these permissions are pretty critical - EKS cannot manage the required AWS services in your account to run the service without them. Would like to hear more about how we can meet your requirements here.
In your regulated environment are you unable to use services like ELB, Autoscaling, Lambda, Dynamo, etc...? These have similar roles that are automatically created by the AWS services.
Like these services, we are working on moving the manual role creation to a Service Linked Role and there are a lot of AWS services that use this IAM model.
@tabern, the permissions are not the issue. The use of AWS Managed Policies is the issue. Some AWS customers do not use AWS Managed Policies because of the risk of possible changes to said policies, not under the customer's control. Instead, they choose to use customer managed policies that reproduce the respective permissions, so that the policy permissions are under control of the customer.
@tabern @jimmyraywv and I work at the same organization and he has captured our concerns. While not a complete blocker to service adoption, the lack of control over the IAM permissions gives some additional process overhead. Service-linked roles help if the customer has control over the role's permissions.
I’m also in a regulated environment where we today still deny use of AWS Managed Policies. EKS (among other new services) causes us troubble with our current regulation. It would be nice to get more understanding of AWS IAM strategies for the future. If AWS Managed Policies is recommended to be used, then we probably have to reevaluate or restrictions to those.
Hi, all - we have been looking for the same (use our policy, not managed policies), and it turns out that while you must use the managed policies to launch the cluster, there's nothing that prohibits you from attaching additional policies that explicitly deny the things you don't want EKS to do.
Great example: we have machinery around automating ELB/ALB/NLB creation and membership and don't use k8s to create load balancers, so we attach a policy to deny EKS anything related to elasticloadbalancing.
I've also created a locked down, experimental policy that blocks pretty much every add/update/delete and have been able to launch a cluster successfully. More testing is needed to see how functional it is, so TBD/YMMV.
@Preskton that's interesting, I don't like giving my EKS nodes access to create load balancers. What are you using for ingress controllers?
@matthewcummings depends on the use case/all the things. At the most basic, we have other tooling manage the xLB and then expose a port (one way or another) that the xLB then load balances across.
You can still use Ingress but delegate the management of the load balancer outside of k8s. Downside: you don't get the one stop shop of declaratively saying "expose this k8s service via a load balancer" in a single manifest.
Key thing is you can use additional policies to block the service role from doing things you don't want it to.
@Preskton thanks for sharing this. This is a pattern that we are seeing customers use on EKS. @ewbankkit does this sound like a solution that would work for you? We could do some work documenting it if yes.
@tabern I think the issue is still not being able to duplicate the necessary policies in non-AWS managed version for policy reasons. Today current managed policies could allow for access to 9 services but tomorrow that could be changed to 11. In a regulated environment that lack of control isn’t an acceptable and therefore would need to be Kubernetes on EC2 like kops vs being able to use EKS.
I went looking for the ability to specify a custom service linked role for ASG when you're using managed node groups and found this issue. It seems like EKS uses Service Linked Roles in a number of places and doesn't let you specify custom roles for any of them.
I've opened a separate issue #1698 to be able to specify a custom service linked role for the ASG that managed node groups create. Issue #1087 also seems related.
Hello @ewbankkit & @jimmyraywv 👋
My organization shares the very same concerns. You described them here perfectly and it's still valid as of 2022.
How did you work around this problem? Could you please tell me about your experiences?
@alicancakil In full transparency, I now work for AWS with the Amazon EKS team. While we haven't changed the need to use AWS Managed Policies for Amazon EKS, we have thoroughly documented the policies, including updates for new and existing policies. This thorough documentation is meant to help organizations better understand the policies and track updates thereof.
https://aws.amazon.com/about-aws/whats-new/2023/10/amazon-eks-customer-managed-iam-policies/
