Support adding/removing inline policy AFTER creating cluster/node-groups

[darkn3rd]

What feature/behavior/change do you want?

Ability to add or remove inline policies to the node role AFTER the cluster has been created would be great. This could be something like:

eksctl utils update-iam-addon-policy --add imageBuilder,externalDNS
eksctl utils update-iam-addon-policy --remove externalDNS

Note that the command line arguments and keys used in the configuration file are inconsistent. I would recommend changing the individual flags to a single flag followed by a comma separated list, e.g. create --iam-addon-policy imageBuilder,externalDNS.

config file withAddonPolicies key	eksctl args
imageBuilder	--full-ecr-access
autoScaler	--asg-access
externalDNS	--external-dns-access
certManager	none
appMesh	--appmesh-access
appMeshPreview	--appmesh-preview-access
ebs	none
fsx	none
efs	none
albIngress	--alb-ingress-access
xRay	none
cloudWatch	none

As an example, using the --external-dns-access for eksctl create cluster (or eksctl create nodegroup) or using externalDNS: true boolean under nodegroups.iam.withAddonPolicies map, the inline policies of PolicyExternalDNSChangeSet and PolicyExternalDNSHostedZones will be added to the node role. There's no way to do this with eksctl AFTER the cluster is created.

Why do you want this feature?

In order to add these inline policies to an existing cluster using eksctl, you have to delete the cluster and recreate it with the option specified. This is a disruptive process

A common use case for this would be the access is needed after, such as image building, the company may decide to switch from Docker or Quay to ECR, so it would be nice to access without destroying the cluster. Another use case would be that during creation of the cluster, adding inline policies seemed like a good idea, such as ExternalDNS access, but as this is too permissive, the company would want to REMOVE it and switch to IRSA.

Both of these are not possible with eksctl. You can do this alternatively through the AWS CLI, or AWS web console, which is complex. For example, to add the inline policies for ExternalDNS after the cluster is created requires doing this:

NODE_GROUP=$(aws eks list-nodegroups --cluster-name $CLUSTER_NAME \
  --query nodegroups --out text)
NODE_ROLE_ARN=$(aws eks describe-nodegroup --cluster-name $CLUSTER_NAME \
  --nodegroup-name $NODE_GROUP --query nodegroup.nodeRole --out text)

cat <<-EOF > PolicyExternalDNSChangeSet.json
{
  "Version": "2012-10-17",
  "Statement": [{
    "Action": ["route53:ChangeResourceRecordSets"],
    "Resource": "arn:aws:route53:::hostedzone/*",
    "Effect": "Allow"
  }]
}
EOF

cat <<-EOF > PolicyExternalDNSHostedZones.json
{
  "Version": "2012-10-17",
  "Statement": [{
    "Action": [
      "route53:ListHostedZones",
      "route53:ListResourceRecordSets",
      "route53:ListTagsForResource"
    ],
    "Resource": "*",
    "Effect": "Allow"
  }]
}
EOF 

POLICIES=(PolicyExternalDNSChangeSet PolicyExternalDNSHostedZones)

# inline policies (same method as ekstcl)
for POLICY in ${POLICIES[*]}; do
  aws iam put-role-policy --role-name $${NODE_ROLE_ARN##*/} \
    --policy-name $POLICY \
    --policy-document file://$POLICY.json
done

# attaching external policies (alternative)
for POLICY in ${POLICIES[*]}; do
  aws iam create-policy --policy-name $POLICY --policy-document file://$POLICY.json
  POLICY_ARN=$(aws iam list-policies --query "Policies[?PolicyName==\`$POLICY\`].Arn" --output text)
  aws iam attach-role-policy --role-name ${NODE_ROLE_ARN##*/} --policy-arn $POLICY_ARN
done

[nikimanoledaki]

Thank you for this feature request! I'm drafting a couple of thoughts around this and will post soon. :)

[nikimanoledaki]

UX-wise, this would be possible through eksctl update nodegroup --config-file, where we would add support for updating the CF template of withAddonPolicies.

We need to revisit the CLI UX for update/upgrade/utils... This is a great example of this UX debate 🤔 If you have any suggestions or preference, we would really appreciate input from the community. Thanks 🙂

For context, we are trying to move away from flags and trying to encourage the use of --config-file. In fact for some features we've dropped the use of flags altogether, especially for the more complex features like IPv6 or Karpenter where a lot of fields are required.