[Bug] Cannot add new addons via config yaml

[skymoore]

What were you trying to accomplish?

Tried adding aws-ebs-csi-driver addon as a new addon to the cluster via the config yaml.

eksctl create addon -f config.yaml

What happened?

The addon was not created. It ignored the new addon.

2022-08-11 16:58:22 [ℹ]  Kubernetes version "1.21" in use by cluster "infra"
2022-08-11 16:58:22 [ℹ]  when creating an addon to replace an existing application, e.g. CoreDNS, kube-proxy & VPC-CNI the --force flag will ensure the currently deployed configuration is replaced
2022-08-11 16:58:22 [ℹ]  no recommended policies found, proceeding without any IAM
2022-08-11 16:58:22 [ℹ]  creating addon

How to reproduce it?

1. create a cluster with:

addons:
  - name: kube-proxy
  - name: coredns
  - name: vpc-cni

2. add a new addon to the cluster config

addons:
  - name: kube-proxy
  - name: coredns
  - name: vpc-cni
  - name: aws-ebs-csi-driver

3. run the command mentioned in the docs against config yaml. eksctl create addon -f config.yaml
4. notice that the additonal addon is not added to the cluster.

Logs

eksctl create addon -f cluster.yaml -v 4 
2022-08-12 09:09:14 [▶]  role ARN for the current session is "arn:aws:sts::REDACTED:assumed-role/AWSReservedSSO_OktaAWSREDACTED/REDACTED"
2022-08-12 09:09:14 [▶]  cluster = &types.Cluster{Arn:(*string)(0x14000a4b580), CertificateAuthority:(*types.Certificate)(0x14000a4b590), ClientRequestToken:(*string)(nil), ConnectorConfig:(*types.ConnectorConfigResponse)(nil), CreatedAt:time.Date(2021, time.November, 22, 21, 10, 11, 187000000, time.UTC), EncryptionConfig:[]types.EncryptionConfig{types.EncryptionConfig{Provider:(*types.Provider)(0x14000a4b4c0), Resources:[]string{"secrets"}, noSmithyDocumentSerde:document.NoSerde{}}}, Endpoint:(*string)(0x14000a4b480), Identity:(*types.Identity)(0x14000a4b550), KubernetesNetworkConfig:(*types.KubernetesNetworkConfigResponse)(0x14000a8ce70), Logging:(*types.Logging)(0x14000a58940), Name:(*string)(0x14000a4b4a0), PlatformVersion:(*string)(0x14000a4b4b0), ResourcesVpcConfig:(*types.VpcConfigResponse)(0x140006a00e0), RoleArn:(*string)(0x14000a4b5b0), Status:"ACTIVE", Tags:map[string]string{"alpha.eksctl.io/cluster-name":"REDACTED", "alpha.eksctl.io/eksctl-version":"0.74.0", "aws:cloudformation:logical-id":"ControlPlane", "aws:cloudformation:stack-id":"arn:aws:cloudformation:us-west-2:REDACTED:stack/REDACTED/REDACTED", "aws:cloudformation:stack-name":"REDACTED", "component":"k8s", "eksctl.cluster.k8s.io/v1alpha1/cluster-name":"REDACTED", "managed-by":"eksctl", "owner":"REDACTED", "part-of":"REDACTED", "tier":"dev", "tracking-url":"REDACTED", "ttl":"1y", "version":"20211122"}, Version:(*string)(0x14000a4b470), noSmithyDocumentSerde:document.NoSerde{}}
2022-08-12 09:09:15 [ℹ]  Kubernetes version "1.21" in use by cluster "REDACTED"
2022-08-12 09:09:15 [ℹ]  when creating an addon to replace an existing application, e.g. CoreDNS, kube-proxy & VPC-CNI the --force flag will ensure the currently deployed configuration is replaced
2022-08-12 09:09:15 [▶]  addon: &{kube-proxy   [] map[]  {false false false false false false false} map[] false}
2022-08-12 09:09:15 [ℹ]  no recommended policies found, proceeding without any IAM
2022-08-12 09:09:15 [ℹ]  creating addon
Error: failed to create addon "kube-proxy": operation error EKS: CreateAddon, https response error StatusCode: 409, RequestID: REDACTED, ResourceInUseException: Addon already exists.
% eksctl get addons --cluster REDACTED
2022-08-12 09:10:31 [ℹ]  Kubernetes version "1.21" in use by cluster "REDACTED"
2022-08-12 09:10:31 [ℹ]  getting all addons
2022-08-12 09:10:33 [ℹ]  to see issues for an addon run `eksctl get addon --name <addon-name> --cluster <cluster-name>`
NAME            VERSION                 STATUS  ISSUES  IAMROLE                                                                         UPDATE AVAILABLE
coredns         v1.8.4-eksbuild.1       ACTIVE  0
kube-proxy      v1.21.2-eksbuild.2      ACTIVE  0                                                                                       v1.21.14-eksbuild.2
vpc-cni         v1.11.2-eksbuild.1      ACTIVE  0       arn:aws:iam::REDACTED:role/REDACTED

Anything else we need to know?

MacOS 12.5 installed eksctl via homebrew named profile, sso via okta Versions

$ eksctl info
eksctl version: 0.107.0-dev+b204c3ce.2022-07-29T12:46:37Z
kubectl version: v1.24.3
OS: darwin

Workaround
install the addon with the command line flags:

eksctl create addon --cluster mycluster --name aws-ebs-csi-driver

[github-actions[bot]]

Hello skymoore :wave: Thank you for opening an issue in eksctl project. The team will review the issue and aim to respond within 1-3 business days. Meanwhile, please read about the Contribution and Code of Conduct guidelines here. You can find out more information about eksctl on our website

[cPu1]

This is indeed a bug in eksctl create addon. Instead of filtering out addons that already exist in the cluster, it's attempting to recreate them, resulting in a 409 error.

Thanks for opening a detailed issue. We'll prioritise fixing this bug.

[skymoore]

This is indeed a bug in eksctl create addon. Instead of filtering out addons that already exist in the cluster, it's attempting to recreate them, resulting in a 409 error.

Thanks for opening a detailed issue. We'll prioritise fixing this bug.

Thank you for fixing this, and the reply. :)

[cPu1]

@skymoore, as a workaround, you can omit the existing addons from ClusterConfig and specify only aws-ebs-csi-driver, and run create addon again.

addons:
  - name: aws-ebs-csi-driver

[skymoore]

@skymoore, as a workaround, you can omit the existing addons from ClusterConfig and specify only aws-ebs-csi-driver, and run create addon again.

addons:
  - name: aws-ebs-csi-driver

@cPu1 Appreciate that. Please also see my workaround above in the issue description. Neither workaround is satisfactory because the idea is infrastructure as code. So If I have to omit stuff that's already represented in the cluster from the code to get it into the desired state, that's not ideal.

[cPu1]

@skymoore, as a workaround, you can omit the existing addons from ClusterConfig and specify only aws-ebs-csi-driver, and run create addon again.

addons:
  - name: aws-ebs-csi-driver

@cPu1 Appreciate that. Please also see my workaround above in the issue description. Neither workaround is satisfactory because the idea is infrastructure as code. So If I have to omit stuff that's already represented in the cluster from the code to get it into the desired state, that's not ideal.

@skymoore, we are going to fix this bug. I suggested a workaround in the meantime for anyone else facing this issue.

[skymoore]

@skymoore, as a workaround, you can omit the existing addons from ClusterConfig and specify only aws-ebs-csi-driver, and run create addon again.

addons:
  - name: aws-ebs-csi-driver

@cPu1 Appreciate that. Please also see my workaround above in the issue description. Neither workaround is satisfactory because the idea is infrastructure as code. So If I have to omit stuff that's already represented in the cluster from the code to get it into the desired state, that's not ideal.

@skymoore, we are going to fix this bug. I suggested a workaround in the meantime for anyone else facing this issue.

@cPu1 Thank you, I was having a bad morning yesterday.

[jagu-sayan]

The work around not working for IPv6 cluster, I got:

Error: the default core addons must be defined for IPv6; missing addon(s): vpc-cni, coredns, kube-proxy

[cPu1]

The work around not working for IPv6 cluster, I got:

Error: the default core addons must be defined for IPv6; missing addon(s): vpc-cni, coredns, kube-proxy

That makes sense, that validation is performed before attempting addon creation. If it's urgent, you can work around this by placing the aws-ebs-csi-driver addon first in the addons list. eksctl should be able to create that addon but the command itself will fail with "Addon already exists" error. But this is a temporary workaround. Please give us some time, we are going to prioritise fixing this bug.