Karpenter is not authorized to perform: ec2:CreateTags on exiting WorkerNodes

[arseny-zinchenko]

Description

I've upgraded the "terraform-aws-modules/eks/aws//modules/karpenter" from 20.0 to 20.24.0, and then upgraded the Helm chart version from 0.37.0 to 1.0.1.

After applying the Helm upgrade, Karpenter's logs constantly throwing the following error for the existing insatnces:

"error":"tagging nodeclaim, tagging instance, UnauthorizedOperation: You are not authorized to perform this operation. User: arn:aws:sts::***:assumed-role/KarpenterIRSA-atlas-eks-ops-1-30-cluster/1724411005093744967 is not authorized to perform: ec2:CreateTags on resource: arn:aws:ec2:us-east-1:***:instance/i-0d715a485281ffc45 because no identity-based policy allows the ec2:CreateTags action.

From the IAM Role, I can that it has the:

			"Action": "ec2:CreateTags",
			"Condition": {
				"StringEquals": {
					"aws:RequestTag/kubernetes.io/cluster/atlas-eks-ops-1-30-cluster": "owned",
					"ec2:CreateAction": [
						"RunInstances",
						"CreateFleet",
						"CreateLaunchTemplate"
					]
				},

If I'm understanding it correctly (and ChatGPT agreed :-) ), the ec2:CreateTags allowed only for new instances, but not for the exiting. And when I'm scaling a Deployment to create new NodeClaims, they are running without any errors from the Karpenter's logs.

• [x] ✋ I have searched the open/closed issues and my issue is not listed.

Versions

• Module version [Required]: 20.24.0
• Terraform version: v1.9.4

Reproduction Code [Required]

The code to deploy Karpenter's module is:

module "karpenter" {
  source = "terraform-aws-modules/eks/aws//modules/karpenter"
  version = "20.24.0"

  cluster_name = module.eks.cluster_name

  irsa_oidc_provider_arn          = module.eks.oidc_provider_arn
  irsa_namespace_service_accounts = ["karpenter:karpenter"]

  create_node_iam_role = false

  node_iam_role_arn         = module.eks.eks_managed_node_groups["${local.env_name_short}-default"].iam_role_arn

  enable_irsa             = true
  create_instance_profile = true

  # backward compatibility with 19.21.0
  # see https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/docs/UPGRADE-20.0.md#karpenter-diff-of-before-v1921-vs-after-v200
  iam_role_name          = "KarpenterIRSA-${module.eks.cluster_name}"
  iam_role_description   = "Karpenter IAM role for service account"
  iam_policy_name        = "KarpenterIRSA-${module.eks.cluster_name}"
  iam_policy_description = "Karpenter IAM role for service account"

  iam_role_use_name_prefix = false

  # already created during EKS 19 > 20 upgrade with 'authentication_mode = "API_AND_CONFIG_MAP"'
  create_access_entry = false
}

[bryantbiggs]

I think this is more of a question for Karpenter or perhaps its called out in the Karpenter upgrade guide for v1.0 - we are simply matching the policy that has been provided by the project

[fraenkel]

Did you set the new var, enable_v1_permissions?

[Mieszko96]

Do you know if it is resolved? i have same issue?

[Mieszko96]

i managed to fix this by update module for karpenter role to 20.24.1 and set enable_v1_permissions to true

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open 30 days with no activity. Remove stale label or comment or this issue will be closed in 10 days

[github-actions[bot]]

This issue was automatically closed because of stale in 10 days

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.