Karpenter Addon: Can't attach custom instanceProfile to Karpenter nodes

[idchrisamzn]

Describe the bug

When attaching a custom instanceProfile to Ec2NodeClassSpec, Karpenter attempts to create the EC2 instances but gets an IAM access denied as it doesn't have iam:PassRole permissions on the custom IAM role in the custom instance profile. The instance profile gets added to default-ec2nodeclass just fine.

It looks like the ec2 IAM role which gets passed to Karpenter's own IAM role is fixed as the one which gets created as part of the addon: https://github.com/aws-quickstart/cdk-eks-blueprints/blob/60256b02db630fe5b422b5e9bf47ded951efe976/lib/addons/karpenter/index.ts#L397 - therefore it could never support a custom instanceProfile

Expected Behavior

Karpenter doesn't create its own EC2 Instance Profile when instanceProfile is specified in the Ec2NodeClassSpec. Instead the custom profile is used to setup Karpenter's own IAM role

Current Behavior

Karpenter attempts to use the custom instance profile but fails to launch due ec2 instances due to the lack of iam:PassRole permissions on the custom IAM role.

Reproduction Steps

Define a custom role as such:

const karpenterRole: iam.RoleProps = {
  assumedBy: new iam.ServicePrincipal('ec2.amazonaws.com'),
  description: 'Role for nodes',
  managedPolicies: [
      { managedPolicyArn: `arn:${Aws.PARTITION}:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly` },
      { managedPolicyArn: `arn:${Aws.PARTITION}:iam::aws:policy/AmazonEKSWorkerNodePolicy` },
      { managedPolicyArn: `arn:${Aws.PARTITION}:iam::aws:policy/AmazonSSMManagedInstanceCore` }
  ] 
};

Use the IAM role properties to create the IAM role and instance profile in the Ec2NodeClassSpec

function returnDefaultEc2NodeClassSpec(clusterName: string): Ec2NodeClassSpec {
  return {
    amiFamily: "Bottlerocket",
    subnetSelectorTerms: [{ tags: { "Name": `${clusterName}-vpc/${clusterName}-vpc/PrivateSubnet*` }}],
    securityGroupSelectorTerms: [{ tags: { "aws:eks:cluster-name": `${clusterName}-ct` }}],
    instanceStorePolicy: 'RAID0',
    tags: { 'name': `${clusterName}-apps`, },
    instanceProfile: blueprints.getResource(context => {
      return new iam.InstanceProfile(context.scope, 'KarpenterInstanceProfile', {
        role: new iam.Role(context.scope, 'KarpenterInstanceRole', karpenterRole)
      });
    }).instanceProfileName,
  };
};

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.147.3 (build 32f0fdb)

EKS Blueprints Version

1.15.1

Node.js Version

v20.15.1

Environment details (OS name and version, etc.)

Ubuntu 22.04.4 LTS

Other information

No response

[idchrisamzn]

Related to #893

[neoakris]

darn thanks, that explains this not working

    const ipv6_support_policy = new iam.PolicyDocument({
        statements: [new iam.PolicyStatement({
            actions: [
              'ec2:AssignIpv6Addresses',
              'ec2:UnassignIpv6Addresses',
            ],
            resources: ['arn:aws:ec2:*:*:network-interface/*'],
        })],
    });
    const karpenter_node_role = new iam.Role(stack, "karpenter-node-role", {
        assumedBy: new iam.ServicePrincipal("ec2.amazonaws.com"),
        managedPolicies: [
            iam.ManagedPolicy.fromAwsManagedPolicyName("AmazonEC2ContainerRegistryReadOnly"),
            iam.ManagedPolicy.fromAwsManagedPolicyName("AmazonEKSWorkerNodePolicy"),
            iam.ManagedPolicy.fromAwsManagedPolicyName("AmazonSSMManagedInstanceCore"),
            iam.ManagedPolicy.fromAwsManagedPolicyName("AmazonEKS_CNI_Policy"),
        ],
        inlinePolicies: {"ipv6_support_for_aws_vpc_cni": ipv6_support_policy},
    });
    const karpenter_node_instance_profile = new iam.InstanceProfile(stack, 'karpenter-node-instance-profile', {
        instanceProfileName: "karpenter-node-instance-profile",
        role: karpenter_node_role,
    });

...
        new blueprints.addons.KarpenterAddOn({
        ...
        ec2NodeClassSpec: {
            ...
            instanceProfile: karpenter_node_instance_profile.instanceProfileName, //ipv6 support
        }
        ...
        });

---

I had trouble with the workaround mentioned there, so gave instance profile a shot, but ran into an error explained by your message above.
kubectl get nodeclaim default-nodepool-vdp26 -o yaml

status:
  conditions:
  - lastTransitionTime: "2024-09-23T22:29:21Z"
    message: 'creating instance, with fleet error(s), UnauthorizedOperation: You are
      not authorized to perform this operation. User: arn:aws:sts::905418347382:assumed-role/dev1-eks-dev1eksblueprintsaddonkarpenterrole4D35444-JgYDRVa6ZcBV/eks-dev1-eks-karpenter--785c1f28-f6e1-42d4-936a-cceecb584919
      is not authorize...'
    reason: LaunchFailed

I guess I'll try another shot at the workaround mentioned on #893