Alignment of IAM permissions with Karpenter versions

[plumdog]

It seems from the main Karpenter project that new versions often tweak the required IAM permissions for the controller.

This is somewhat related to #231.

These permissions haven't changed much from 1.0.0 to current latest 1.5.0, but have changed a little (I'm aware some of the lines shown as changed here are inconsequential):

$ getKarpenterControllerPolicyDocument() { version="$1"; curl -s "https://raw.githubusercontent.com/aws/karpenter/v$version/website/content/en/preview/getting-started/getting-started-with-karpenter/cloudformation.yaml" | yq '.Resources.KarpenterControllerPolicy.Properties.PolicyDocument' }
$ diff <(getKarpenterControllerPolicyDocument "1.0.0") <(getKarpenterControllerPolicyDocument "1.5.0")                                                                                                                                    11c11,12
<         "arn:${AWS::Partition}:ec2:${AWS::Region}:*:subnet/*"
---
>         "arn:${AWS::Partition}:ec2:${AWS::Region}:*:subnet/*",
>         "arn:${AWS::Partition}:ec2:${AWS::Region}:*:capacity-reservation/*"
44c45,46
<         "arn:${AWS::Partition}:ec2:${AWS::Region}:*:spot-instances-request/*"
---
>         "arn:${AWS::Partition}:ec2:${AWS::Region}:*:spot-instances-request/*",
>         "arn:${AWS::Partition}:ec2:${AWS::Region}:*:capacity-reservation/*"
137c139
<         "ec2:DescribeAvailabilityZones",
---
>         "ec2:DescribeCapacityReservations",
182c184,187
<           "iam:PassedToService": "ec2.amazonaws.com"
---
>           "iam:PassedToService": [
>             "ec2.amazonaws.com",
>             "ec2.amazonaws.com.cn"
>           ]

But have changed substantially since 0.32.0, which is the Karpenter version on which this project bases the IAM permissions granted to the controller, see https://github.com/aws-samples/cdk-eks-karpenter/blob/main/src/index.ts#L456-L459 and see:

$ diff <(getKarpenterControllerPolicyDocument "0.32.0") <(getKarpenterControllerPolicyDocument "1.5.0")
5c5
<       "Sid": "AllowScopedEC2InstanceActions",
---
>       "Sid": "AllowScopedEC2InstanceAccessActions",
10d9
<         "arn:${AWS::Partition}:ec2:${AWS::Region}:*:spot-instances-request/*",
13c12
<         "arn:${AWS::Partition}:ec2:${AWS::Region}:*:launch-template/*"
---
>         "arn:${AWS::Partition}:ec2:${AWS::Region}:*:capacity-reservation/*"
20a20,36
>       "Sid": "AllowScopedEC2LaunchTemplateAccessActions",
>       "Effect": "Allow",
>       "Resource": "arn:${AWS::Partition}:ec2:${AWS::Region}:*:launch-template/*",
>       "Action": [
>         "ec2:RunInstances",
>         "ec2:CreateFleet"
>       ],
>       "Condition": {
>         "StringEquals": {
>           "aws:ResourceTag/kubernetes.io/cluster/${ClusterName}": "owned"
>         },
>         "StringLike": {
>           "aws:ResourceTag/karpenter.sh/nodepool": "*"
>         }
>       }
>     },
>     {
28c44,46
<         "arn:${AWS::Partition}:ec2:${AWS::Region}:*:launch-template/*"
---
>         "arn:${AWS::Partition}:ec2:${AWS::Region}:*:launch-template/*",
>         "arn:${AWS::Partition}:ec2:${AWS::Region}:*:spot-instances-request/*",
>         "arn:${AWS::Partition}:ec2:${AWS::Region}:*:capacity-reservation/*"
37c55,56
<           "aws:RequestTag/kubernetes.io/cluster/${ClusterName}": "owned"
---
>           "aws:RequestTag/kubernetes.io/cluster/${ClusterName}": "owned",
>           "aws:RequestTag/eks:eks-cluster-name": "${ClusterName}"
52c71,72
<         "arn:${AWS::Partition}:ec2:${AWS::Region}:*:launch-template/*"
---
>         "arn:${AWS::Partition}:ec2:${AWS::Region}:*:launch-template/*",
>         "arn:${AWS::Partition}:ec2:${AWS::Region}:*:spot-instances-request/*"
57a78
>           "aws:RequestTag/eks:eks-cluster-name": "${ClusterName}",
80a102,104
>         "StringEqualsIfExists": {
>           "aws:RequestTag/eks:eks-cluster-name": "${ClusterName}"
>         },
82a107
>             "eks:eks-cluster-name",
114c139
<         "ec2:DescribeAvailabilityZones",
---
>         "ec2:DescribeCapacityReservations",
148d172
<         "sqs:GetQueueAttributes",
156c180
<       "Resource": "arn:${AWS::Partition}:iam::${AWS::AccountId}:role/KarpenterNodeRole-${ClusterName}",
---
>       "Resource": "${KarpenterNodeRole.Arn}",
160c184,187
<           "iam:PassedToService": "ec2.amazonaws.com"
---
>           "iam:PassedToService": [
>             "ec2.amazonaws.com",
>             "ec2.amazonaws.com.cn"
>           ]
167c194
<       "Resource": "*",
---
>       "Resource": "arn:${AWS::Partition}:iam::${AWS::AccountId}:instance-profile/*",
173a201
>           "aws:RequestTag/eks:eks-cluster-name": "${ClusterName}",
184c212
<       "Resource": "*",
---
>       "Resource": "arn:${AWS::Partition}:iam::${AWS::AccountId}:instance-profile/*",
192a221
>           "aws:RequestTag/eks:eks-cluster-name": "${ClusterName}",
204c233
<       "Resource": "*",
---
>       "Resource": "arn:${AWS::Partition}:iam::${AWS::AccountId}:instance-profile/*",
223c252
<       "Resource": "*",
---
>       "Resource": "arn:${AWS::Partition}:iam::${AWS::AccountId}:instance-profile/*",

I think this project should do (at least) one of the following:

• support a range of recent Karpenter versions and ensure that the right IAM permissions are granted based on the version chosen (either by granting the combined permissions needed for all of them, or by granting specific permissions correct for each particular version)
• support a way for users of this project to assign updated permissions based on the Karpenter version they have picked, eg by removing the use of private for methods and attributes related to these permissions, and documenting a way to introduce additional permissions.

[andskli]

I think allowing for users to bring their own permissions is a good starting point. Useful and allows for opting in.

Ideally, I would like to be in a place where we dynamically generate the IAM permission code for each release of Karpenter. I think maintaining it by hand over time will be hard to do/keep correct.