[Bug] eksctl does not use correct K8S credentials in post-creation operations

[otterley]

In certain situations, eksctl fails to perform certain operations, such as cluster and nodegroup deletion, after cluster creation. When this happens, eksctl fails to authenticate to the EKS cluster. Here's an example:

2023-09-21 14:02:52 [ℹ]  deleting EKS cluster "ec2-performance-workshop"
2023-09-21 14:02:53 [ℹ]  will drain 0 unmanaged nodegroup(s) in cluster "ec2-performance-workshop"
2023-09-21 14:02:53 [ℹ]  starting parallel draining, max in-flight of 1
2023-09-21 14:02:53 [✖]  Node group drain failed: %!w(*errors.StatusError=&{{{ } {   <nil>} Failure Unauthorized Unauthorized <nil> 401}})
Error: Unauthorized

(Note: the error message should have more cleanly rendered instead of the%!w(...) text, but that's an issue for a different time.)

The situation in which this occurs is as follows:

1. Cluster is created in AWS CodeBuild container having an IAM role that gives it access to create and manipulate EKS clusters using the AWS APIs.
2. An IAM Role is created by CloudFormation. This Role is given system:masters permission to access the cluster via the iamIdentityMappings section of the eksctl configuration file. This Role has full administrative access to the cluster after creation time, but no AWS API privileges.
3. Some time later, and in a different context (e.g. developer laptop), the user wants to delete the cluster. She has already run aws eks update-kubeconfig, so her ~/.kube/config file already has the appropriate user/exec section in it that calls aws eks get-token. She also has AWS credentials that allow her full access to the AWS account, but her principal is not the IAM role used for EKS cluster access. She then invokes eksctl delete-cluster, which fails with the above error.

The underlying cause seems to be that, in the deletion context, eksctl is using AWS credentials for authentication for all purposes instead of using two sources of truth:

1. ~/.kube/config as a source of truth for K8S authentication information, and
2. AWS credentials for AWS API authentication.

What appears to be happening is that eksctl is translating the existing AWS credentials (which map to her AWS administrative principal) to a K8S credential, but since that principal has no privileges in the cluster, the Unauthorized error is returned.

If eksctl used ~/.kube/config as a source of truth (using the standard BuildConfigFromFlags Go client API), then this problem would be resolved.

Alternatively, if the user could pass multiple IAM role ARNs to eksctl - one mapping to the principal used for K8S operations, and another to the principal used for AWS operations - it could mitigate this issue.

[a-hilaly]

Thank you @otterley for reporting this! I'm not super familiar with auth components for eksctl, but i'll have a closer look soon and see if i can come with a solution/PR.

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.