[BUG][Nodegroup] Deletion of cluster with Spot Managed Nodegroup results in Role deletion error

[youngjeong46]

I tried deleting a cluster with the following managed nodegroups:

{
          id: "mng-1",
          minSize: 1,
          maxSize: 5,
          desiredSize: 2,
          amiType: eks.NodegroupAmiType.AL2_X86_64,
          instanceTypes: [new ec2.InstanceType('m5.2xlarge')],
          nodeGroupCapacityType: eks.CapacityType.ON_DEMAND,
        },
        {
          id: "spot-1",
          instanceTypes: [
            new ec2.InstanceType('t2.xlarge'),
            new ec2.InstanceType('t3.xlarge'),
            new ec2.InstanceType('t3.small'),
          ],
          nodeGroupCapacityType: eks.CapacityType.SPOT,
        }

When deleting a cluster, the IAM Role for Spot Nodegroup cannot be deleted due to attached policy: AmazonSSMManagedInstanceCore

[shapirov103]

It is a known issue for all Isengard accounts and occurs due to Account Guardian onboarding and PVRE stacks. The account protection automatically injects the policy to the node group. This change is viewed by CFN as external, hence requiring to manually address (prevents deletion). There are two things we can do:

1. Workaround: add the ssm agent add-on to your stack and it should take care of the issue.
2. We can apply a shim, which will automatically add the aforementioned policy to the node group role.

Since both of these are specific to Isengard, I suggest we move forward with 1/ and create a shim for development purposes repurposing this issue. Could be a good "start" issue for new contributors.

[ohschmidty]

I am seeing this beyond isengard and am receiving errors from CloudFormation attempting to delete NodeGroup Roles similar to this: eksblueprinttestingNodegroupeksblueprintsmngngNodeGroupRole26A8E201

[shapirov103]

@daveschmidt86 if your example was leveraging an account onboarded to the Account Guardian (which is also the case for all (most) Isengard accounts) then this issue will happen. Other than the approach outlined in my previous comment, there is not much we can do atm.

@youngjeong46 please confirm the workaround works. Closing soon.

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open 60 days with no activity. Remove stale label or comment or this issue will be closed in 10 days

[github-actions[bot]]

Issue closed due to inactivity.