Allow to pass `bootstrapSelfManagedAddons` argument to cluster

[ahanoff]

Hello!

• Vote on this issue by adding a 👍 reaction
• If you want to implement this feature, comment to let us know (we'll work with you on design, scheduling, etc.)

Issue details

Currently EKS cluster sets by default bootstrapSelfManagedAddons as true which can not be overridden. That setting creates 3 addons: vpc-cni, coredns, and kube-proxy

As a workaround users can use the defaultAddonsToRemove input property with ["vpc-cni", "coredns", and "kube-proxy"] to remove all addons on cluster creation.

Affected area/feature

Prevents from creating clean cluster without any addon which is beneficial if you use custom CNI like Cilium

[t0yv0]

Hi @ahanoff thanks for the issue. Have you tried the defaultAddonsToRemove property?

https://www.pulumi.com/registry/packages/eks/api-docs/cluster/#defaultaddonstoremove_nodejs

I believe if you populate that property with a list of "vpc-cni", "coredns", "kube-proxy" then they should not be auto-created anymore.

[ahanoff]

Have you tried the defaultAddonsToRemove property?

hi @t0yv0, thanks! Yes, that's what I currently set while waiting for this issue resolution

List of addons to remove upon creation. Any addon listed will be "adopted" and then removed

Actually I'm confused with two things:

• defaultAddonsToRemove comment above as I'm not sure what adopted means in this case. If it means it will be installed first and then removed, would it be a case when AWS VPC CNI leaves some iptables rules that needs to be cleared?
• having two options (bootstrapSelfManagedAddons and defaultAddonsToRemove) to do same things?

Anyway I'm going test your suggestion this week and get back with feedback

[t0yv0]

Great feedback, let me double-check and see if we can clarify the docstrings.

[flostadler]

Hey @ahanoff, defaultAddonsToRemove wouldn't leave any ip tables rules behind because the adopt->delete procedure is done before there's any nodes in the cluster. In detail, we're creating an EKS Addon for every addon you mention in defaultAddonsToRemove. This adopts the existing resources on the cluster into Pulumi's management. Finally, we're deleting those addons again, which cleans up the underlying resources.

But I agree with you that having two options to achieve the same goal is not ideal! pulumi-eks inherits this option from the pulumi-aws provider that manages the underlying cluster resource. defaultAddonsToRemove was added to that provider before AWS added the bootstrapSelfManagedAddons feature. I created an issue in pulumi-aws to track unifying this situation: https://github.com/pulumi/pulumi-aws/issues/4755

[ahanoff]

have you tried the defaultAddonsToRemove property?

hi @t0yv0, @flostadler, I've tested this and would like to confirm that addons are removed and Cilium CNI installed passing cilium connectivity test successfully

[flostadler]

Awesome, thanks @ahanoff. I'll keep the issue open to track this enhancement once we tackled https://github.com/pulumi/pulumi-aws/issues/4755.

I also updated the description with the defaultAddonsToRemove workaround

[lbi22]

Hello, sorry if I jump into the conversation, but I have an issue that is related, but not exactly the same.

As the property bootstrapSelfManagedAddons is not managed by eks.Cluster, I have an issue since I imported the cluster. The problem is that my original cluster had this property set to False, and as this property in aws.eks.Cluster triggers the recreation of it, it defeats the purpose of importing the cluster.

Do you have any idea how I could avoid this issue? I tried to use the ignoreChanges, but I don't know how to set it up to ignore specific properties of child resources.