[EKS] [request]: Manage IAM identity cluster access with EKS API

[ayosec]

Tell us about your request

CloudFormation resources to register IAM roles in the aws-auth ConfigMap.

Which service(s) is this request for? EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?

A Kubernetes cluster managed by EKS is able to authenticate users with IAM roles. This is very useful to grant access to Lambda functions. However, as described in the documentation, every IAM role has to be registered manually in a ConfigMap with the name aws-auth.

For every IAM role we add to the CloudFormation stack, we have to add an entry like this:

mapRoles: |
  - rolearn: "arn:aws:iam::11223344:role/stack-FooBarFunction-AABBCCDD"
    username: lambdafoo
    groups:
      - system:masters
  - ...

This process is a bit tedious, and it is hard to automate.

It will be much better if those IAM roles can be registered directly in the CloudFormation template. For example, with something like this:

LambdaKubeUser:
  Type: AWS::EKS::MapRoles::Entry
  Properties:
    Cluster: !Ref EKSCluster
    RoleArn: !GetAtt FunctionRole.Arn
    UserName: lambdafoo
    Groups:
      - system:masters

Thus, CloudFormation will add and remove entries in the ConfigMap as necessary, with no extra manual steps.

Another AWS::EKS::MapUsers::Entry can be used to register IAM users in mapUsers.

With this addition, we can automate the extra step to register the IAM role of the worker nodes when a new EKS instance is created:

NodeInstanceKubeUser:
  Type: AWS::EKS::MapRoles::Entry
  Properties:
    Cluster: !Ref EKSCluster
    RoleArn: !GetAtt NodeInstanceRole.Arn
    UserName: system:node:{{EC2PrivateDNSName}}
    Groups:
      - system:bootstrappers
      - system:nodes

[abelmokadem]

@ayosec have you created something to automate this as of now? I'm running into this when setting up a cluster using CloudFormation. Do you mind sharing your current approach?

[ayosec]

have you created something to automate this as of now?

Unfortunately, no. I haven't found a reliable way to do it 100% automatic.

Do you mind sharing your current approach?

My current approach is to generate the ConfigMap using a template:

1. All relevant ARNs are available in the outputs of the stack.
2. A Ruby script reads those outputs, and fills a template.
3. Finally, the generated YAML is applied with kubectl apply -f -.

[dardanbekteshi]

Adding this feature on CloudFormation would allow the same feature to be added on AWS CDK. This will greatly simplify the process of adding/removing new nodes, for example.

[willejs]

I also thought about this. An api to manage the config map for aws-iam-authenticator is interesting, i think would be a bit clunky. I am using terraform to create an eks cluster, and this approach is alot nicer https://github.com/terraform-aws-modules/terraform-aws-eks/pull/355

[inductor]

I'd love this

[schlomo]

Anybody from AWS care to comment on this feature request?

[mikestef9]

With the release of Managed Nodes with CloudFormation support, EKS now automatically handles updating aws-auth config map for joining nodes to a cluster.

Does this satisfy the initial use case here, or is there a separate ask to manage adding users to the aws-auth config map via CloudFormation?

[inductor]

@mikestef9 I think that https://github.com/aws/containers-roadmap/issues/554 can be one of the similar issues why you would like this kind of options

[ayosec]

@mikestef9

Does this satisfy the initial use case here, or is there a separate ask to manage adding users to the aws-auth config map via CloudFormation?

My main use case is with Lambda functions.

The managed nodes feature is pretty cool, and very useful for new EKS clusters, but most of our modifications to the aws-auth ConfigMap are to add or remove roles for Lambda functions.

[tnh]

@mikestef9 It would be useful to then allow people / roles to be able to then run kubectl commands.

Right now, we have a CI deploy role - but we want to allow other saml based users to be able to kubectl

We do a post cluster creation to kubectl

apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-auth
  namespace: kube-system
data:
  mapRoles: |
    - rolearn: {{ ClusterAdminRoleArn }}
      username: system:node:{{ '{{EC2PrivateDNSName}}' }}
      groups:
        - system:bootstrappers
        - system:nodes
    - rolearn: arn:aws:iam::{{ AccountId }}:role/{{ AdminRoleName }}
      username: admin
      groups:
        - system:masters
    - rolearn: arn:aws:iam::{{ AccountId }}:role/{{ CIRoleName }}
      username: ci
      groups:
        - system:masters
    - rolearn: arn:aws:iam::{{ AccountId }}:role/{{ ViewRoleName }}
      username: view

But I'd much rather have this configMAp created by me during the cluster creation

[nckturner]

@mikestef9 Some relevant issues related to EKS users debugging authentication problems (https://github.com/kubernetes-sigs/aws-iam-authenticator/issues/174 and https://github.com/kubernetes-sigs/aws-iam-authenticator/issues/275) that imo are data points in favor of API and Cloudformation management of auth mappings (and configurable admin role: https://github.com/aws/containers-roadmap/issues/554).

[nemo83]

This ^^, how can we get this implemented? Can anyone from AWS tell us if they gonna support this at CF template level? Or a workaround is needed at eksctl level?

[inductor]

@nemo83 AWS team tagged researching this issue, so that's the stage of this issue.

[nicorikken]

I'm also looking into automating the updates to this configmap from cloudformation. Doing so via lambda seems doable.

My main concern with automation are race-conditions on the contents of the configmap when applying updates as the content has to be parsed. A strategic merge is not possible. If the configuration would be implemented in one or more (one per entry) CRD's it would be easier to apply a patch. In that case existing efforts on Kubernetes support for CloudFormation like kubernetes-resources-provider can be reused.

Update: we gave up on writing a lambda to update the configmap. The code became too complex and fragile. We now template it separately.

Update 2: I had a concern for automatically updating the configmap if it would become corrupt and thereby prevent API access. With the current workings of AWS (1 sept 2020) there is a way of recovering from an aws-auth configmap corruption:

aws-auth configmap recovery (tested 1 sept 2020)

The prerequisite is to have a pod in the cluster running with a serviceaccount that can update the aws-auth configmap. Ideally something that you can interact with, like k8s-dashboard or in our case ArgoCD.

Then if the aws-auth become corrupt you can hopefully still update the configmap that way.

If that is not the case because the nodes have lost their access we can use the EKS-managed Node Group to restore node access to the Kubernetes API. You can create an EKS-managed Node Group of just 1 node with the role that is also used by your cluster nodes. (Note: this is not recommended by AWS, but we abuse AWS's power to update the configmap on the managed master nodes.)

AWS will now add this role to the aws-auth configmap:

apiVersion: v1
kind: ConfigMap
metadata:
  annotations:
  name: aws-auth
  namespace: kube-system
data:
  mapRoles: |
    ...
    # this entry is added by AWS automically
    - rolearn: < NodeInstanceRole ARN >
      username: system:node:{{EC2PrivateDNSName}}
      groups:
      - system:bootstrappers
      - system:nodes

Deletion of that Node Group will remove that entry (for which AWS warns you), so the serviceaccount access is required to ensure another method of cluster access, like via the kubectl CLI. Update to aws-auth configmap to get that method of access. Then the Node Group can be removed, which in turn removes the added aws-auth configmap entry that was automatically created earlier. Now the persistent connection (e.g. kubectl CLI) can be used to permanently fix the configmap to ensure the nodes have access.

⚠ Note: if a service is automatically but incorrectly updating the configmap it would be harder, of not impossible to recover. ⚠

[hellupline]

I would go a extra mile and ask AWS to create an API to manage aws-auth, with IAM action associated

in case I delete the IAM role/user associated with the cluster creation ( detail: this user/role is not visible after, you have to save this info outside the cluster, or tag the cluster with it. )

and if I dont add another admin to the cluster, I am now locked out of the cluster,

for me, this is a major issue, because I use federated auth, users ( and my day-to-day account ) are efemeral... my user can be recreated without warning with another name/ID,

the ideia is: can AWS add an IAM action like ESHttpGet/ESHttpPost ? ( example from ElasticSearch, because is a third party software )

[mikestef9]

Hey @hellupline

We are actually working on exactly that right now, an EKS API to manage IAM users and their permissions to an EKS cluster. This will allow you to manage IAM users via IaC tools like CloudFormation

[yanivpaz]

@mikestef9 how it's going to be different compared to https://github.com/aws-quickstart/quickstart-amazon-eks/blob/main/templates/amazon-eks-controlplane.template.yaml#L109

[markussiebert]

I wonder, why this isn't possible with eks clusters (but with selfhosted k8s clusters on AWS?)

https://github.com/kubernetes-sigs/aws-iam-authenticator#crd-alpha

Even looking at the cdk implementation of auth mapping, it would be simple to get rid of some limitations that exist right now (stack barrier, imported clusters ...)

So if something like CF Support for Auth-Mapping will be implemented (i support this) it would be good, if it won't conflict with the crd's I hope coming soon to eks.

[gp42]

Hey @hellupline

We are actually working on exactly that right now, an EKS API to manage IAM users and their permissions to an EKS cluster. This will allow you to manage IAM users via IaC tools like CloudFormation

Any news on this issue?

[vennemp]

For those wanting to know, this is currently supported via aws cdk - they create a custom resource which invokes a lambda function, and updates the config map. A little hacky and glad I didn't have to write that code but it works like charm!

[dany74q]

@vennemp - Cool ! From what I gather though, that's only possible if it makes sense to assign said lambda the cluster-creation role; This might be problematic if the clusters were created by a 'plain' user, or by a highly-privileged role, which would be risky to assign.

I think that the 'lock-in' to the cluster creation role is problematic - especially if clusters are created in a non-centralized fashion; it would be awesome to have the ability to control permissions in AWS-alone - along w/ predefined roles one could grant (a-la Kubernetes Viewer / Editor / Owner)

This would also give one the ability to grant said roles to third parties for cluster-auditing, introspection, or just as an auth mechanism - that would work cluster-wide and not require a per-cluster tweak.

[vennemp]

@dany74q yes that is what it does and i agree wholeheartedly. It’s definitely a pain. Adding the ability to update the Config mp at launch to include other principals would be the best way One thing I’ve done is edit the trust relationship of the role used to create the cluster to include a Trust relationship with another principal - like ec2 or an aws account or another role(you can have more than one!). So I can use the role arn attribute in aws eks update-kubeconfig Command. That way I can just assume that original role.

[mikestef9]

@dany74q

"it would be awesome to have the ability to control permissions in AWS-alone - along w/ predefined roles one could grant (a-la Kubernetes Viewer / Editor / Owner)"

This is exactly the API we are building that I referenced in a previous comment. We'll have a handful of pre-baked polices (Admin, Editor, Viewer) that you can attach through the API at a cluster wide or specific namespace level, without ever having to add anything in RBAC. You can still of course use RBAC if you need to define more fine-grained permissions per IAM entity.

[tsndqst]

@mikestef9 is there a separate issue in the roadmap for the API you mentioned or should we follow this issue for updates on that API?

[mikestef9]

Let's use this issue, I updated the title

[manuelcoppotelli]

The aws-iam-authenticator has implemented the IAMIdentityMappings CRD.

May be a good idea to follow along with this solution and implement the same!

[gunzy83]

Going to drop my $0.02 here and share the use case. We are rebuilding our IAM across all of our accounts with AWS SSO as the entry point to our accounts (console and CLI) so we don't have credentials to distribute anymore. As a part of that, we are trying to follow recommendations for everything we do (least privilege, etc) and Cloudformation service roles are a part of that.

I was talking with our SA contact from AWS about how the role that creates the cluster is the only principal that can perform the update to the config map but if that is a role we pass to Cloudformation when we are automatically deploying our entire stack, we don't want to allow our CI nodes to assume role to that role, update kubeconfig and be able to do everything system:masters can do (mind you it can now, we want to do better).

Our SA pointed me to https://github.com/aws-samples/eks-aws-auth-configmap as a starting point to create a custom resource and allow the lambda to assume the Cloudformation service role. This is better but still not ideal as the service role would have permission to create roles (with a boundary policy), access the KMS key created in Cloudformation, and more; quite a bit more permission than required.

I really hope this will be implemented in the API and eventually added to Cloudformation.

[adammw]

@mikestef9 is there further work ongoing on a different API for managing IAM identities, or is the IAMIdentityMappings CRD support essentially delivering this API change and there will just be other UI changes to ease the management of these resources?

reason I ask is that is eksctl work on any IAM identity mapping features is stalled until the new API design is announced: https://github.com/weaveworks/eksctl/issues/874

[mikestef9]

It's a totally separate EKS API - not related to the IAMIdentityMappings CRD

[dany74q]

Hey @mikestef9 ! Wondered if there's an update on this, this is one of the most exiting features to be implemented in EKS, for our use cases at least - appreciating the complexity of implementing it at scale, a rough estimation (months / quarters away) would be amazing.

Thanks a ton !