eksRole doesnt have proper permissions

[orcutt989]

What happened?

clusterName-eksCluster (eks:index:Cluster$aws:eks/cluster:Cluster)
error: 1 error occurred:
	* error creating EKS Cluster (clusterName-eksCluster-102f259): InvalidParameterException: Role with arn: arn:aws:iam::awsAccountNumber:role/clusterName-eksRole-role-d5b33dc, could not be assumed because it does not exist or the trusted entity is not correct
{
  RespMetadata: {
    StatusCode: 400,
    RequestID: "db5a6cd2-542d-4343-9145-f671fd7f97c7"
  },
  ClusterName: "clusterName-eksCluster-102f259",
  Message_: "Role with arn: arn:aws:iam::awsAccountNumber:role/clusterName-eksRole-role-d5b33dc, could not be assumed because it does not exist or the trusted entity is not correct"
}

Starting up a new cluster returns this error. Seems like the eksRole doesnt have the permissions in the newest version of pulumi-eks. v41.2 possibly

Steps to reproduce

pulumi up with new eks cluster definition and pulumi-eks v41.2

Expected Behavior

Cluster stands up without error.

Actual Behavior

clusterName-eksCluster (eks:index:Cluster$aws:eks/cluster:Cluster)
error: 1 error occurred:
	* error creating EKS Cluster (clusterName-eksCluster-102f259): InvalidParameterException: Role with arn: arn:aws:iam::awsAccountNumber:role/clusterName-eksRole-role-d5b33dc, could not be assumed because it does not exist or the trusted entity is not correct
{
  RespMetadata: {
    StatusCode: 400,
    RequestID: "db5a6cd2-542d-4343-9145-f671fd7f97c7"
  },
  ClusterName: "clusterName-eksCluster-102f259",
  Message_: "Role with arn: arn:aws:iam::awsAccountNumber:role/clusterName-eksRole-role-d5b33dc, could not be assumed because it does not exist or the trusted entity is not correct"
}

Versions used

"@pulumi/awsx": "0.40.0", "@pulumi/eks": "0.41.2", "@pulumi/pulumi": "3.36.0", "@pulumi/aws": "^5.0.0",

Additional context

@harryttd

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[harryttd]

Related: https://github.com/weaveworks/eksctl/issues/2182 By opening up all IAM permissions for the user deploying and creating the cluster that allowed this to work. I don't understand why. And i don't know the minimum required perms for IAM that would be sufficient.

[stack72]

Hi @orcutt989

Can you give me a snippet of code that recreates this issue please?

Paul

[gunzy83]

I ran into this error message but during an upgrade operation. The issue is that the role was updated by someone previously only has AmazonEKSClusterPolicy but also needs AmazonEKSServicePolicy as the cluster was originally created a long time ago. This commit breaks clusters that were created a long time ago and upgraded over time but I am unsure why this would affect create operations.

[macropin]

We don't use pulumi, however we've experienced the same error when upgrading some clusters. So far we have not been able to determine the cause, but suspect it is an AWS bug which may be related to the EKS platform version.

Like @gunzy83 , adding AmazonEKSServicePolicy to the cluster role resolves the issue. And when reviewing the policy/service usage it appears that AmazonEKSServicePolicy is granting access to CloudWatch Logs.

tldr probably an AWS EKS bug.

[mikhailshilkov]

Since we haven't heard back, I don't think it's actionable for us. I'll go ahead and close this issue as stale. Please open a new issue if the problem still persists and provide the details for us to reproduce it. Thank you!