containers-roadmap
containers-roadmap copied to clipboard
aws
[EKS]: IAM Roles for Service Accounts enhancements for usage across multiple clusters
Community Note
- Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
- Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment
Tell us about your request IAM Roles for Service Accounts (IRSA) enables you to associate an IAM role with a Kubernetes service account, and follow the principle of least privilege by giving pods only the AWS API permissions they need, without sharing permissions to all pods running on the same node. This feature works well with a smaller number of clusters, but becomes more difficult to manage as the number of EKS clusters grows, notably:
- Needing to create an IAM OIDC provider for each cluster
- Updating the trust policy of every IAM role needed by that cluster with cluster’s OIDC provider URL that maps to a particular namespace and service account name.
- Given the 2048 character limit of IAM trust policies, only around 5 clusters can be trusted per role.
- After 5 clusters, you need to duplicate IAM roles to be used by additional clusters.
- Coordination between devs and admins - Development teams that own IAM roles often need to re-run Terraform apply or CloudFormation scripts to update trust policies for new clusters.
Given these pain points, EKS is considering a change to the way IRSA works, moving credential vending to the EKS control plane (similar to how ECS and Lambda works). With this change, a trust policy would only need to be updated once to trust a service principal like eks-pods.amazonaws.com, then you would call an EKS API to provide the IAM role to service account mapping, ex.
aws eks associate-role \
--cluster-name $CLUSTER_NAME \
--role-arn $ROLE_ARN \
--namespace $KUBERNETES_NAMESPACE \
--service-account $SERVICE_ACCOUNT
We are looking for your feedback on this proposal, and to hear any additional pain points encountered with IRSA today that would not be solved by such a solution.
Are you currently working around this issue? Creating and managing duplicate roles to be used across multiple clusters
This would be very useful as it is definitely a problem we run into.
Would this work cross-account? We often associate service accounts with roles in accounts other than the one in which EKS is running.
@mikestef9 thanks for raising this.
When running large clusters/platforms on top of AWS and requiring IAM there are a few more constraints than just the trust relationship payload size.
-
Each AWS account has a hard limit of 5000 IAM roles per account. With the above it would be ideal to not be bound by a single accounts hard limit and to allow cross account assume roles. Running a multi-tenant workload can quite easily consume this limit with a few thousand deployments.
-
Do you have more details around the implementation of this? E.g. Would this require aws-sdk changes to the credential provider chain and therefore a bump for all tenants using AWS SDKs?
-
How would you limit the trust relationship to a single or group of clusters?
-
Would there be any caveats/gotchas that would need to consider as part of this proposal?
@gregoryfranklin yes we plan to make cross account roles work with this proposal.
@ajohnstone we haven't settled on implementation yet. One idea is to use a CSI driver to retrieve a pod’s OIDC token from the Kubelet and exchange the token for IAM Role credentials from EKS. Another is a new credential provider in all AWS SDKs, but that would be hampered by requiring all clients to update.
You manage trust through the EKS API now. The IAM trust policy only is updated once to trust the EKS service principal.
This might solve a challenge I am currently having with IRSA. There are situations where I have to replace an existing cluster with a new one and then shift traffic to the new cluster, either due to disaster recovery or a change which cant be applied without downtime. Currently this would require all developers to update their trust policy with the new OIDC provider, so that the applications can authenticate from the new cluster.
Would this new solution require all roles to be associated with the new cluster or could that be solved somehow when replacing a cluster?
You would still need to call the EKS API to add the mapping, but this can be automated using IaC tools as part of the new cluster creation process. There would be no need to update a role trust policy when creating a new cluster, and you no longer run into trust policy character limits when you need a role associated with many clusters.
It sounds like we would still have the same issues as we do today. In my scenario individual development teams setup their own resources and IAM roles with Terraform, this is a separate state from the one which creates the cluster. The manifests are deployed with FluxCD into the cluster. This enables us to replace a cluster over night without any development team being the wiser as the new cluster will pull down the same manifests as the old one does. The issue here today is that each development team would have to update the OIDC provider in their trust policy.
Correct me if I am wrong but it sounds like there is still a need in the new proposed solution for each team to associate their IAM roles with service accounts in the new cluster. Basically keeping the same issue that we have today but solved with a different IaC resource.
Correct me if I am wrong but it sounds like there is still a need in the new proposed solution for each team to associate their IAM roles with service accounts in the new cluster. Basically keeping the same issue that we have today but solved with a different IaC resource.
If teams are creating the IAM roles with terraform, can they not lookup the cluster id using a data source?
We store the current list of OIDC providers in an SSM parameter when we create a cluster. Terraform is therefore able to lookup the appropriate IDs in order to generate the policy. The same appears to be true with the new proposal, but instead of needing to template the trust policy there would just be a new resource type to create the mapping. The cluster name parameter would change each time the cluster is re-built but its a lot easier to just lookup the value for that single parameter than template the trust policy.
With this proposal, yes there would be an EKS API call to make for each cluster instead of a trust policy update. Additionally, there would no longer be a need to annotate a service account with the IAM role Arn.
So is what you are looking for more along the lines of some way to trust an IAM role for all clusters in an account, without having to specify each cluster?
@mikestef9 yeah that is basically it. I do not want to have to inform development teams to rerun their Terraform if I replace the cluster. It would be nice to either configure trust to an account of some sort of regex pattern for the cluster name.
If a CSI Driver is used, I hope it's not a daemonset. We're quickly running into issues with the number of daemonsets we need to run on nodes that consume resources we can then not use to run workloads (currently at about 7 per node). At large cluster sizes, this is not insignificant where you might end up with 10k workload pods and 28k daemonset pods. If using the vpc-cni-plugin and IP allocations out of a VPC, it's even worse since now you're consuming potentially scarce IPs.
We've also run into the trust policy size limits already.
Talking from personal experience of dealing with both the AssumeRoleWithWebIdentity & AWS SSO default credential provider chain updates to multiple language SDKs: It is a nice long term change but actually doing it was an absolute nightmare. None of the AWS SDK teams coordinate their work when service teams add new methods leaving all customers in the lurch to deal with it. When you are dealing with 100s of devs and multiple languages this was a giant time sink that AWS was not helpful with.
This is a must. In the way it is right now, 1 policy ~400bytes, assume role max policy size 4096bytes. 9clusters are enough to reach this limit.
- IAM Variable interpolations for EKS/OIDC --> inexistent
- Length of the ARN -> Gigantic
- Length of OIDC URL -> Gigantic
IMHO, i can't label the today integration.. where EKS elements are treat as a foreign element of AWS as "production-grade". Even if we produce 1 role per cluster, we will hit max roles... or max policies, or max attachments..
Does anyone knows a workaround, besides producing resource roles (nodes, cluster, etc) where everyone can access everyone else AWS resources.. ??
Do we have an ETA for this feature ? we kinda struggle..
Just faced the same issue. We do run multiple clusters, each in their own AWS account. Currently we use KIAM to assume roles for our production services. The KIAM servers of all clusters/account currently runs on a special node which attaches a instance role of one central account, so the product teams specify only this account to allow assuming the role and we can scale out new clusters without any IAM change.
I just stopped a initiative to migrate all our services to the new IRSA as the effort to migrate all assume role policies is too high (additional to the AWS SDK library updates...). We use cloudformation across the place, so its a nightmare to get all our product teams on board to update their roles.
Considering possible Disaster Recovery cases when i need to re-create everything or even deploy into another region would not be possible as we cannot assume the roles anymore :disappointed:, there is just no good way to keep the roles synced with possible OIDC provider changes.
We're running into this as well using ClusterAPI + AWS integration to spin up clusters - the ability to use a regex or wildcard to allow clusters to use existing IRSA roles would be great.
Yes we also just ran head first into the 4096 character IAM policy size limit trying to scale out by adding more clusters. And we're basically stuck right now until we can do something about this.
My team uses ClusterAPI to manage clusters in AWS right now, and we've been discussing an option that does not require any changes to IRSA which some here may find helpful. The general idea is to run all the clusters under a single trust domain or identity provider. The big requirement is that you ensure that namespaces are managed the same way in all clusters under the trust domain. There will be no way for IRSA to distinguish between same named namespaces in clusters under the trust domain.
Configure the clusters to use the same service account issuer. You have the option to also make them all use the same signing key, but that's not a requirement. The service account issuer string needs to be a URL that can serve the OIDC discovery documents publicly, such as S3. You also need to set the service account jwks URI to point into the same location. This will ensure that the .well-known/openid-configuration file generated by the cluster is correctly configured to point to the openid/v1/jwks file.
Extract the discovery documents from the clusters. The .well-known/openid-configuration file should be the same for all of them. Merge the openid/v1/jwks files together. Then copy them to S3 or wherever they need to be hosted to satisfy the issuer URL. In order to merge the jwks files, concatenate the arrays at .keys together and emit the result as JSON. Note that as you add clusters to and remove clusters from the trust domain, the jwks file will need to be updated accordingly, either adding or removing keys from the array. We're still working out exactly how we plan to do this via automation.
Merging the jwks files is only necessary if you continue use cluster specific signing keys, which we recommend since that allows you to evict individual signing keys from the trust domain without affecting other clusters. If you use the same signing key for all clusters though, then you need only copy the OIDC discovery documents from one of the clusters to the hosting solution since all clusters will have identical documents. You also never need to update the jwks file again as you add and remove clusters under the trust domain. Buyer beware on this!
Once all of that is set up, you can define a single IAM OIDC provider to represent the trust domain. Point it to the endpoint you set up for hosting the OIDC discovery documents. Now create IAM roles as usual using the new provider. All workloads running in clusters under the same trust domain will be able to use the roles as if they run under a single cluster. Note the point about namespace handling mentioned earlier.
It would be great if EKS allowed the service account issuer to be specified because then this same solution could be used.
I share the same concerns that @phillebaba has expressed: even following IaC best practices updating all the roles trust policies after that a new cluster is provisioned can be pretty complex at scale. As others have stated before, this is a pretty big limitation for disaster recovery/migration scenarios and also for the overall scalability of the IRSA system.
EKS clusters should be able to rely on a shared OIDC provider:
Having a shared identity provider would allow active-active or active-passive type cluster deployments to be provisioned
transparently while being able to enforce tier/env based trust policie (eg. prod - test - dev ) without the current scalability issues.
We are in the same situation as @phillebaba and @GaruGaru and would love for it to be possible for a given SA in any cluster in the account to assume the IRSA role automatically, whether that is using a shared OIDC provider, or a more flexible (wildcard) trust policy syntax. It's crucial to be able to do blue/green cluster failovers without a highly coupled, hard-to-scale IaC setup.
The solution shared by @javanthropus seems to solve the problem as it’s been stated, albeit with a extra few steps such as creating the .well-known/openid-configuration and associated jwks, and then building a method of retrieving the public key half of the service account signing key from EKS control planes to place in the jwks if you want a unique signing key per cluster (which is optimal from a security and management perspective).
Currently I’m not entirely sure it is possible to get the public key portion of the signing key from EKS control planes, or using user-managed keys as SA token signing keys. Enabling the retrieval of that information via the AWS API would make this solution possible today.
For reference, the pod-identity-webhook repository does document all the steps needed for creating a provider independently of an EKS cluster. I also have a POC repository demonstrating how to generate a signing keypair and use it as the signing key, as well as some IaC to leverage this setup (granted, this was done using RKE2, not EKS, because there’s no obvious way to get the public key from EKS).
With this proposal, yes there would be an EKS API call to make for each cluster instead of a trust policy update. Additionally, there would no longer be a need to annotate a service account with the IAM role Arn.
@mikestef9 Any chance you'd be ready and willing to share some more details/design around how this is going to work ?
With this proposal, yes there would be an EKS API call to make for each cluster instead of a trust policy update. Additionally, there would no longer be a need to annotate a service account with the IAM role Arn.
Instead of calling the EKS API (We have 30+ roles per cluster), can we keep the annotation for the same effect?
Instead of calling the EKS API (We have 30+ roles per cluster), can we keep the annotation for the same effect?
This would have some potentially problematic security implications wouldn't it? If you have permission within kubernetes to modify service accounts in any namespace in any cluster, you could grant access to assume that role. Something outside the cluster needs to be able to restrict by cluster/namespace at least.
Instead of calling the EKS API (We have 30+ roles per cluster), can we keep the annotation for the same effect?
This would have some potentially problematic security implications wouldn't it? If you have permission within kubernetes to modify service accounts in any namespace in any cluster, you could grant access to assume that role. Something outside the cluster needs to be able to restrict but cluster/namespace at least.
Fair point, I hadn't thought of that, just thinking of a way to automate this for all our regional clusters
I'm curious if the new EKS Pod Identity Associations are intended to be part of the solution/alternative for this.
- EKS Pod Identity Assocations
- https://github.com/eksctl-io/eksctl/pull/7312
As a result, IAM roles no longer need to reference an OIDC provider and hence won't be tied to a single cluster anymore.
Is it worth running yet another daemonset I wonder? 🤔
Behind the scenes, the implementation of pod identity associations is running an agent as a daemonset on the worker nodes.
And no support for cross-account roles likely requiring application code changes.
Closing this issue since IAM Roles for Service Accounts enhancements was launched yesterday. Please see announcement details -
Please feel free to open a new issue if you have feedback/enhancement requests for the newly launched EKS Pod Identity feature.
From the docs it is not really clear to me if the pod identity feature is gonna work in a cross-account scenario. Did someone test it already or can give more clarity here?
From the docs it is not really clear to me if the pod identity feature is gonna work in a cross-account scenario. Did someone test it already or can give more clarity here?
Thank you for the feedback. EKS Pod Identity works in cross account scenario. We are in the process of updating EKS user guide to give more clarity on this. We are targeting for the updates to be available in user guide next week or so.