[Bug] propagateASGTags option not working

[YDKK]

What were you trying to accomplish?

Attempted to automatically apply label and taint tags to ASGs using the propagateASGTags option. I expected the following tags to be created to the ASGs automatically:

• k8s.io/cluster-autoscaler/node-template/label/my-cool-label: pizza
• k8s.io/cluster-autoscaler/node-template/taint/feaster: "true:NoExecute"

What happened?

The ASGs created are not tagged for labels and taints.

How to reproduce it?

Create a cluster using the following configuration and eksctl create cluster -f test.yaml -v 4 command.

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: test-cluster2
  region: ap-northeast-1
  version: "1.22"

managedNodeGroups:
  - name: test-group
    amiFamily: Bottlerocket
    instanceType: t3.small
    desiredCapacity: 0
    minSize: 0
    maxSize: 1
    iam:
      withAddonPolicies:
        autoScaler: true
    labels:
      my-cool-label: pizza
    taints:
      - key: feaster
        value: "true"
        effect: NoExecute
    propagateASGTags: true

Logs

2022-06-14 16:04:57 [▶]  role ARN for the current session is "arn:aws:iam::***:user/***"
2022-06-14 16:04:57 [ℹ]  eksctl version 0.101.0
2022-06-14 16:04:57 [ℹ]  using region ap-northeast-1
2022-06-14 16:04:57 [▶]  determining availability zones
2022-06-14 16:04:57 [ℹ]  setting availability zones to [ap-northeast-1d ap-northeast-1c ap-northeast-1a]
2022-06-14 16:04:57 [▶]  VPC CIDR (192.168.0.0/16) was divided into 8 subnets [192.168.0.0/19 192.168.32.0/19 192.168.64.0/19 192.168.96.0/19 192.168.128.0/19 192.168.160.0/19 192.168.192.0/19 192.168.224.0/19]
2022-06-14 16:04:57 [ℹ]  subnets for ap-northeast-1d - public:192.168.0.0/19 private:192.168.96.0/19
2022-06-14 16:04:57 [ℹ]  subnets for ap-northeast-1c - public:192.168.32.0/19 private:192.168.128.0/19
2022-06-14 16:04:57 [ℹ]  subnets for ap-northeast-1a - public:192.168.64.0/19 private:192.168.160.0/19
2022-06-14 16:04:57 [ℹ]  nodegroup "test-group" will use "" [Bottlerocket/1.22]
2022-06-14 16:04:57 [ℹ]  using Kubernetes version 1.22
2022-06-14 16:04:57 [ℹ]  creating EKS cluster "test-cluster2" in "ap-northeast-1" region with managed nodes
2022-06-14 16:04:57 [▶]  cfg.json = \
{
    "kind": "ClusterConfig",
    "apiVersion": "eksctl.io/v1alpha5",
    "metadata": {
        "name": "test-cluster2",
        "region": "ap-northeast-1",
        "version": "1.22"
    },
    "iam": {
        "withOIDC": false,
        "vpcResourceControllerPolicy": true
    },
    "vpc": {
        "cidr": "192.168.0.0/16",
        "subnets": {
            "private": {
                "ap-northeast-1a": {
                    "az": "ap-northeast-1a",
                    "cidr": "192.168.160.0/19"
                },
                "ap-northeast-1c": {
                    "az": "ap-northeast-1c",
                    "cidr": "192.168.128.0/19"
                },
                "ap-northeast-1d": {
                    "az": "ap-northeast-1d",
                    "cidr": "192.168.96.0/19"
                }
            },
            "public": {
                "ap-northeast-1a": {
                    "az": "ap-northeast-1a",
                    "cidr": "192.168.64.0/19"
                },
                "ap-northeast-1c": {
                    "az": "ap-northeast-1c",
                    "cidr": "192.168.32.0/19"
                },
                "ap-northeast-1d": {
                    "az": "ap-northeast-1d",
                    "cidr": "192.168.0.0/19"
                }
            }
        },
        "manageSharedNodeSecurityGroupRules": true,
        "autoAllocateIPv6": false,
        "nat": {
            "gateway": "Single"
        },
        "clusterEndpoints": {
            "privateAccess": false,
            "publicAccess": true
        }
    },
    "privateCluster": {
        "enabled": false,
        "skipEndpointCreation": false
    },
    "managedNodeGroups": [
        {
            "name": "test-group",
            "amiFamily": "Bottlerocket",
            "instanceType": "t3.small",
            "desiredCapacity": 0,
            "minSize": 0,
            "maxSize": 1,
            "volumeSize": 80,
            "ssh": {
                "allow": false
            },
            "labels": {
                "alpha.eksctl.io/cluster-name": "test-cluster2",
                "alpha.eksctl.io/nodegroup-name": "test-group",
                "my-cool-label": "pizza"
            },
            "privateNetworking": false,
            "tags": {
                "alpha.eksctl.io/nodegroup-name": "test-group",
                "alpha.eksctl.io/nodegroup-type": "managed"
            },
            "iam": {
                "withAddonPolicies": {
                    "imageBuilder": false,
                    "autoScaler": true,
                    "externalDNS": false,
                    "certManager": false,
                    "appMesh": null,
                    "appMeshPreview": null,
                    "ebs": false,
                    "fsx": false,
                    "efs": false,
                    "awsLoadBalancerController": false,
                    "albIngress": false,
                    "xRay": false,
                    "cloudWatch": false
                }
            },
            "securityGroups": {
                "withShared": null,
                "withLocal": null
            },
            "volumeType": "gp3",
            "volumeName": "/dev/xvdb",
            "volumeIOPS": 3000,
            "volumeThroughput": 125,
            "propagateASGTags": true,
            "disableIMDSv1": false,
            "disablePodIMDS": false,
            "instanceSelector": {},
            "bottlerocket": {
                "settings": {}
            },
            "taints": [
                {
                    "key": "feaster",
                    "value": "true",
                    "effect": "NoExecute"
                }
            ],
            "releaseVersion": ""
        }
    ],
    "availabilityZones": [
        "ap-northeast-1d",
        "ap-northeast-1c",
        "ap-northeast-1a"
    ]
}

2022-06-14 16:04:57 [ℹ]  1 nodegroup (test-group) was included (based on the include/exclude rules)
2022-06-14 16:04:57 [ℹ]  will create a CloudFormation stack for cluster itself and 0 nodegroup stack(s)
2022-06-14 16:04:57 [ℹ]  will create a CloudFormation stack for cluster itself and 1 managed nodegroup stack(s)
2022-06-14 16:04:57 [ℹ]  if you encounter any issues, check CloudFormation console or try 'eksctl utils describe-stacks --region=ap-northeast-1 --cluster=test-cluster2'
2022-06-14 16:04:57 [ℹ]  Kubernetes API endpoint access will use default of {publicAccess=true, privateAccess=false} for cluster "test-cluster2" in "ap-northeast-1"
2022-06-14 16:04:57 [ℹ]  CloudWatch logging will not be enabled for cluster "test-cluster2" in "ap-northeast-1"
2022-06-14 16:04:57 [ℹ]  you can enable it with 'eksctl utils update-cluster-logging --enable-types={SPECIFY-YOUR-LOG-TYPES-HERE (e.g. all)} --region=ap-northeast-1 --cluster=test-cluster2'
2022-06-14 16:04:57 [ℹ]
2 sequential tasks: { create cluster control plane "test-cluster2",
    2 sequential sub-tasks: {
        wait for control plane to become ready,
        2 sequential sub-tasks: {
            create managed nodegroup "test-group",
            propagate tags to ASG for managed nodegroup "test-group",
        },
    }
}
2022-06-14 16:04:57 [▶]  started task: create cluster control plane "test-cluster2"
2022-06-14 16:04:57 [ℹ]  building cluster stack "eksctl-test-cluster2-cluster"
2022-06-14 16:04:57 [▶]  CreateStackInput = &cloudformation.CreateStackInput{StackName:(*string)(0xc000849a30), Capabilities:[]types.Capability{"CAPABILITY_IAM"}, ClientRequestToken:(*string)(nil), DisableRollback:(*bool)(0xc000bf6b50), EnableTerminationProtection:(*bool)(nil), NotificationARNs:[]string(nil), OnFailure:"", Parameters:[]types.Parameter(nil), ResourceTypes:[]string(nil), RoleARN:(*string)(nil), RollbackConfiguration:(*types.RollbackConfiguration)(nil), StackPolicyBody:(*string)(nil), StackPolicyURL:(*string)(nil), Tags:[]types.Tag{types.Tag{Key:(*string)(0xc000122e10), Value:(*string)(0xc000122e20), noSmithyDocumentSerde:document.NoSerde{}}, types.Tag{Key:(*string)(0xc000122e40), Value:(*string)(0xc000122e50), noSmithyDocumentSerde:document.NoSerde{}}, types.Tag{Key:(*string)(0xc000122e70), Value:(*string)(0xc000122e80), noSmithyDocumentSerde:document.NoSerde{}}}, TemplateBody:(*string)(0xc00088f390), TemplateURL:(*string)(nil), TimeoutInMinutes:(*int32)(nil), noSmithyDocumentSerde:document.NoSerde{}}
2022-06-14 16:04:58 [ℹ]  deploying stack "eksctl-test-cluster2-cluster"
2022-06-14 16:05:28 [ℹ]  waiting for CloudFormation stack "eksctl-test-cluster2-cluster"
2022-06-14 16:05:58 [ℹ]  waiting for CloudFormation stack "eksctl-test-cluster2-cluster"
2022-06-14 16:06:58 [ℹ]  waiting for CloudFormation stack "eksctl-test-cluster2-cluster"
2022-06-14 16:07:58 [ℹ]  waiting for CloudFormation stack "eksctl-test-cluster2-cluster"
2022-06-14 16:08:58 [ℹ]  waiting for CloudFormation stack "eksctl-test-cluster2-cluster"
2022-06-14 16:09:58 [ℹ]  waiting for CloudFormation stack "eksctl-test-cluster2-cluster"
2022-06-14 16:10:58 [ℹ]  waiting for CloudFormation stack "eksctl-test-cluster2-cluster"
2022-06-14 16:11:58 [ℹ]  waiting for CloudFormation stack "eksctl-test-cluster2-cluster"
2022-06-14 16:12:59 [ℹ]  waiting for CloudFormation stack "eksctl-test-cluster2-cluster"
2022-06-14 16:13:59 [ℹ]  waiting for CloudFormation stack "eksctl-test-cluster2-cluster"
2022-06-14 16:14:59 [ℹ]  waiting for CloudFormation stack "eksctl-test-cluster2-cluster"
2022-06-14 16:15:59 [ℹ]  waiting for CloudFormation stack "eksctl-test-cluster2-cluster"
2022-06-14 16:15:59 [▶]  processing stack outputs
2022-06-14 16:15:59 [▶]  completed task: create cluster control plane "test-cluster2"
2022-06-14 16:15:59 [▶]  started task:
    2 sequential sub-tasks: {
        wait for control plane to become ready,
        2 sequential sub-tasks: {
            create managed nodegroup "test-group",
            propagate tags to ASG for managed nodegroup "test-group",
        },
    }

2022-06-14 16:15:59 [▶]  started task: wait for control plane to become ready
2022-06-14 16:15:59 [▶]  started task: wait for control plane to become ready
2022-06-14 16:18:00 [▶]  cluster = &types.Cluster{Arn:(*string)(0xc000abbc40), CertificateAuthority:(*types.Certificate)(0xc000abbc20), ClientRequestToken:(*string)(nil), ConnectorConfig:(*types.ConnectorConfigResponse)(nil), CreatedAt:time.Date(2022, time.June, 14, 7, 5, 43, 440000000, time.UTC), EncryptionConfig:[]types.EncryptionConfig(nil), Endpoint:(*string)(0xc000abbc60), Identity:(*types.Identity)(0xc000abbbf0), KubernetesNetworkConfig:(*types.KubernetesNetworkConfigResponse)(0xc000b30600), Logging:(*types.Logging)(0xc000ace7a0), Name:(*string)(0xc000abbc80), PlatformVersion:(*string)(0xc000abbcc0), ResourcesVpcConfig:(*types.VpcConfigResponse)(0xc000a30700), RoleArn:(*string)(0xc000abbca0), Status:"ACTIVE", Tags:map[string]string{"Name":"eksctl-test-cluster2-cluster/ControlPlane", "alpha.eksctl.io/cluster-name":"test-cluster2", "alpha.eksctl.io/eksctl-version":"0.101.0", "aws:cloudformation:logical-id":"ControlPlane", "aws:cloudformation:stack-id":"arn:aws:cloudformation:ap-northeast-1:***:stack/eksctl-test-cluster2-cluster/4c5a23b0-ebb0-11ec-ad66-06bf774ae855", "aws:cloudformation:stack-name":"eksctl-test-cluster2-cluster", "eksctl.cluster.k8s.io/v1alpha1/cluster-name":"test-cluster2"}, Version:(*string)(0xc000abbc50), noSmithyDocumentSerde:document.NoSerde{}}
2022-06-14 16:18:00 [▶]  completed task: wait for control plane to become ready
2022-06-14 16:18:00 [▶]  completed task: wait for control plane to become ready
2022-06-14 16:18:00 [▶]  started task:
    2 sequential sub-tasks: {
        create managed nodegroup "test-group",
        propagate tags to ASG for managed nodegroup "test-group",
    }
2022-06-14 16:18:00 [▶]  waiting for 1 parallel tasks to complete
2022-06-14 16:18:00 [▶]  started task:
    2 sequential sub-tasks: {
        create managed nodegroup "test-group",
        propagate tags to ASG for managed nodegroup "test-group",
    }
2022-06-14 16:18:00 [▶]  waiting for 1 parallel tasks to complete
2022-06-14 16:18:00 [▶]  started task:
    2 sequential sub-tasks: {
        create managed nodegroup "test-group",
        propagate tags to ASG for managed nodegroup "test-group",
    }

2022-06-14 16:18:00 [▶]  started task: create managed nodegroup "test-group"
2022-06-14 16:18:00 [ℹ]  building managed nodegroup stack "eksctl-test-cluster2-nodegroup-test-group"
2022-06-14 16:18:00 [▶]  CreateStackInput = &cloudformation.CreateStackInput{StackName:(*string)(0xc0009d4560), Capabilities:[]types.Capability{"CAPABILITY_IAM"}, ClientRequestToken:(*string)(nil), DisableRollback:(*bool)(0xc0006ec1a8), EnableTerminationProtection:(*bool)(nil), NotificationARNs:[]string(nil), OnFailure:"", Parameters:[]types.Parameter(nil), ResourceTypes:[]string(nil), RoleARN:(*string)(nil), RollbackConfiguration:(*types.RollbackConfiguration)(nil), StackPolicyBody:(*string)(nil), StackPolicyURL:(*string)(nil), Tags:[]types.Tag{types.Tag{Key:(*string)(0xc000122e10), Value:(*string)(0xc000122e20), noSmithyDocumentSerde:document.NoSerde{}}, types.Tag{Key:(*string)(0xc000122e40), Value:(*string)(0xc000122e50), noSmithyDocumentSerde:document.NoSerde{}}, types.Tag{Key:(*string)(0xc000122e70), Value:(*string)(0xc000122e80), noSmithyDocumentSerde:document.NoSerde{}}, types.Tag{Key:(*string)(0xc000a79180), Value:(*string)(0xc000a791b0), noSmithyDocumentSerde:document.NoSerde{}}, types.Tag{Key:(*string)(0xc000a79270), Value:(*string)(0xc000a792b0), noSmithyDocumentSerde:document.NoSerde{}}}, TemplateBody:(*string)(0xc000a792c0), TemplateURL:(*string)(nil), TimeoutInMinutes:(*int32)(nil), noSmithyDocumentSerde:document.NoSerde{}}
2022-06-14 16:18:00 [ℹ]  deploying stack "eksctl-test-cluster2-nodegroup-test-group"
2022-06-14 16:18:00 [ℹ]  waiting for CloudFormation stack "eksctl-test-cluster2-nodegroup-test-group"
2022-06-14 16:18:31 [ℹ]  waiting for CloudFormation stack "eksctl-test-cluster2-nodegroup-test-group"
2022-06-14 16:19:15 [ℹ]  waiting for CloudFormation stack "eksctl-test-cluster2-nodegroup-test-group"
2022-06-14 16:19:55 [ℹ]  waiting for CloudFormation stack "eksctl-test-cluster2-nodegroup-test-group"
2022-06-14 16:19:55 [▶]  processing stack outputs
2022-06-14 16:19:55 [▶]  completed task: create managed nodegroup "test-group"
2022-06-14 16:19:55 [▶]  started task: propagate tags to ASG for managed nodegroup "test-group"
2022-06-14 16:19:55 [▶]  completed task: propagate tags to ASG for managed nodegroup "test-group"
2022-06-14 16:19:55 [▶]  completed task:
    2 sequential sub-tasks: {
        create managed nodegroup "test-group",
        propagate tags to ASG for managed nodegroup "test-group",
    }

2022-06-14 16:19:55 [▶]  completed task:
    2 sequential sub-tasks: {
        create managed nodegroup "test-group",
        propagate tags to ASG for managed nodegroup "test-group",
    }
2022-06-14 16:19:55 [▶]  completed task:
    2 sequential sub-tasks: {
        create managed nodegroup "test-group",
        propagate tags to ASG for managed nodegroup "test-group",
    }
2022-06-14 16:19:55 [▶]  completed task:
    2 sequential sub-tasks: {
        wait for control plane to become ready,
        2 sequential sub-tasks: {
            create managed nodegroup "test-group",
            propagate tags to ASG for managed nodegroup "test-group",
        },
    }

2022-06-14 16:19:55 [ℹ]  waiting for the control plane availability...
2022-06-14 16:19:56 [▶]  merging kubeconfig files
2022-06-14 16:19:56 [▶]  setting current-context to ***@test-cluster2.ap-northeast-1.eksctl.io
2022-06-14 16:19:56 [✔]  saved kubeconfig as "C:\\Users\\***\\.kube\\config"
2022-06-14 16:19:56 [ℹ]  no tasks
2022-06-14 16:19:56 [▶]  no actual tasks
2022-06-14 16:19:56 [✔]  all EKS cluster resources for "test-cluster2" have been created
2022-06-14 16:19:56 [▶]  kubectl: "C:\\Program Files\\Docker\\Docker\\resources\\bin\\kubectl.exe"
2022-06-14 16:19:57 [▶]  kubectl version: v1.24.0
2022-06-14 16:19:57 [▶]  found authenticator: aws
2022-06-14 16:19:59 [ℹ]  kubectl command should work with "C:\\Users\\***\\.kube\\config", try 'kubectl get nodes'
2022-06-14 16:19:59 [✔]  EKS cluster "test-cluster2" in "ap-northeast-1" region is ready
2022-06-14 16:19:59 [▶]  cfg.json = \
{
    "kind": "ClusterConfig",
    "apiVersion": "eksctl.io/v1alpha5",
    "metadata": {
        "name": "test-cluster2",
        "region": "ap-northeast-1",
        "version": "1.22"
    },
    "iam": {
        "serviceRoleARN": "arn:aws:iam::***:role/eksctl-test-cluster2-cluster-ServiceRole-FCY3X3FAG8P5",
        "withOIDC": false,
        "vpcResourceControllerPolicy": true
    },
    "vpc": {
        "id": "vpc-0079f4c110b29aefb",
        "cidr": "192.168.0.0/16",
        "securityGroup": "sg-0ffb0a8612d4c3625",
        "subnets": {
            "private": {
                "ap-northeast-1a": {
                    "id": "subnet-0f748e1dae8099529",
                    "az": "ap-northeast-1a",
                    "cidr": "192.168.160.0/19"
                },
                "ap-northeast-1c": {
                    "id": "subnet-04513eab92f87cbe8",
                    "az": "ap-northeast-1c",
                    "cidr": "192.168.128.0/19"
                },
                "ap-northeast-1d": {
                    "id": "subnet-0299b6adadd4eb5ec",
                    "az": "ap-northeast-1d",
                    "cidr": "192.168.96.0/19"
                }
            },
            "public": {
                "ap-northeast-1a": {
                    "id": "subnet-00e12906014cb7f63",
                    "az": "ap-northeast-1a",
                    "cidr": "192.168.64.0/19"
                },
                "ap-northeast-1c": {
                    "id": "subnet-02a97fb688448ddb8",
                    "az": "ap-northeast-1c",
                    "cidr": "192.168.32.0/19"
                },
                "ap-northeast-1d": {
                    "id": "subnet-016923d7d216c2449",
                    "az": "ap-northeast-1d",
                    "cidr": "192.168.0.0/19"
                }
            }
        },
        "sharedNodeSecurityGroup": "sg-0fa8b3fe6164d3312",
        "manageSharedNodeSecurityGroupRules": true,
        "autoAllocateIPv6": false,
        "nat": {
            "gateway": "Disable"
        },
        "clusterEndpoints": {
            "privateAccess": false,
            "publicAccess": true
        }
    },
    "privateCluster": {
        "enabled": false,
        "skipEndpointCreation": false
    },
    "managedNodeGroups": [
        {
            "name": "test-group",
            "amiFamily": "Bottlerocket",
            "instanceType": "t3.small",
            "desiredCapacity": 0,
            "minSize": 0,
            "maxSize": 1,
            "volumeSize": 80,
            "ssh": {
                "allow": false
            },
            "labels": {
                "alpha.eksctl.io/cluster-name": "test-cluster2",
                "alpha.eksctl.io/nodegroup-name": "test-group",
                "my-cool-label": "pizza"
            },
            "privateNetworking": false,
            "tags": {
                "alpha.eksctl.io/nodegroup-name": "test-group",
                "alpha.eksctl.io/nodegroup-type": "managed"
            },
            "iam": {
                "withAddonPolicies": {
                    "imageBuilder": false,
                    "autoScaler": true,
                    "externalDNS": false,
                    "certManager": false,
                    "appMesh": null,
                    "appMeshPreview": null,
                    "ebs": false,
                    "fsx": false,
                    "efs": false,
                    "awsLoadBalancerController": false,
                    "albIngress": false,
                    "xRay": false,
                    "cloudWatch": false
                }
            },
            "securityGroups": {
                "withShared": null,
                "withLocal": null
            },
            "volumeType": "gp3",
            "volumeName": "/dev/xvdb",
            "volumeIOPS": 3000,
            "volumeThroughput": 125,
            "propagateASGTags": true,
            "disableIMDSv1": false,
            "disablePodIMDS": false,
            "instanceSelector": {},
            "bottlerocket": {
                "settings": {
                    "kubernetes": {}
                }
            },
            "taints": [
                {
                    "key": "feaster",
                    "value": "true",
                    "effect": "NoExecute"
                }
            ],
            "releaseVersion": ""
        }
    ],
    "availabilityZones": [
        "ap-northeast-1d",
        "ap-northeast-1c",
        "ap-northeast-1a"
    ]
}

Anything else we need to know?

Windows 11 Pro 21H2 Build 22000.675 Used scoop command for install eksctl. Default AWS profile.

Versions

$ eksctl info
eksctl version: 0.101.0
kubectl version: v1.24.0
OS: windows

[github-actions[bot]]

Hello YDKK :wave: Thank you for opening an issue in eksctl project. The team will review the issue and aim to respond within 1-3 business days. Meanwhile, please read about the Contribution and Code of Conduct guidelines here. You can find out more information about eksctl on our website

[youwalther65]

I faced the same issue $ eksctl info eksctl version: 0.101.0 kubectl version: v1.23.5 OS: linux

on AWS EKS 1.22

[Skarlso]

Thanks for the issue. Will verify.

[matti]

same here with:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: 'test-1'
  region: 'eu-north-1'

managedNodeGroups:
  - name: 'test-16-32-6-2022-06-16-14-12-45'
    labels:
      test: 'yes'
    volumeType: 'gp3'
    volumeSize: 6
    volumeIOPS: 3000
    volumeThroughput: 125
    minSize: 0
    maxSize: 450
    spot: true
    tags:
      'k8s.io/cluster-autoscaler/node-template/label/test': 'yes'
    iam:
      withAddonPolicies:
        autoScaler: true
        imageBuilder: true
        ebs: true
        albIngress: true
    instanceTypes:
      - c5.4xlarge
      - c5a.4xlarge
    ami: 'ami-0078ce758e0c53482'

I don't really understand the codebase, but it looks like https://github.com/weaveworks/eksctl/pull/5286/files this doesn't address the propagation which is now broken/missing

[matti]

so workaround like this:

nodegroup=$1

# fetch nodegroups asg
asg=$(
  aws eks describe-nodegroup --region $REGION --cluster-name $CLUSTER_NAME \
    --nodegroup-name $nodegroup \
    --query "nodegroup.resources.autoScalingGroups[0].name" \
    --output text
)

# fetch all labels from nodegroup
labels=$(
  aws eks describe-nodegroup --region $REGION --cluster-name $CLUSTER_NAME \
    --nodegroup-name $nodegroup \
    --query "nodegroup.labels" \
    --output=json | jq -r 'to_entries[] | .key + "=" + .value'
)


# copy every label except those which are `alpha.eksctl.io...`
for label in $labels; do
  label_key=${label%=*}
  label_value=${label#*=}

  case $label_key in
    alpha.eksctl.io*)
      echo "skip: $label"
      continue
    ;;
  esac

  echo "tag: $label"

  aws autoscaling create-or-update-tags --region $REGION \
    --tags "ResourceId=${asg},ResourceType=auto-scaling-group,Key=k8s.io/cluster-autoscaler/node-template/label/${label_key},Value=${label_value},PropagateAtLaunch=true"
done

[Skarlso]

The code propagates TAGS. Not labels or taints. So if you want tags to be propagated to the ASG you need to define tags.

  tags:
    k8s.io/cluster-autoscaler/node-template/label/my-cool-label: pizza

[Skarlso]

Please close if this answers your question. Thanks!

[matti]

the problem is that it doesn't propagate these tags to the underlying auto scaling group - they only get correctly added to the eks tags listing what is shown in the screenshot above.

So the tags are kinda worthless because they are not pushed to the auto scaling group

this was already asked in #4965 which was closed, please the comment https://github.com/weaveworks/eksctl/issues/4965#issuecomment-1070699110

which says "So yeah, this ticket isn't a duplicate, it's what #3793 (comment) was actually requesting."

So it was also requested in that #3793, but closed.

so to reiterate: eks nodegroup tags need to be propagated all the way to the ASG, now they are only set to eks nodegroups, but not to the underlying ASGs. please see my script above how I workaround this.

[Skarlso]

That screenshot is from the ASG.

[matti]

okay, I'll try to make my own screenshots soon

[youwalther65]

ok, so this extension of eksctl doesn't deliver what we hoped, right?

But I run into another thing when trying to follow the docs not sure what I am doing wrong:

...
managedNodeGroups:
  - name: ng-spot
    amiFamily: AmazonLinux2
    instanceSelector:
      vCPUs: 2
      memory: "4" # 4 GiB, unit defaults to GiB
      cpuArchitecture: x86_64 # default value
    minSize: 1
    desiredCapacity: 2
    maxSize: 2
    volumeSize: 100
    volumeType: gp3
    volumeEncrypted: true
    privateNetworking: true
    iam:
      attachPolicyARNs:
        - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
        - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
        - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
        - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
    spot: true
    labels:
      my-cool-label: pizza
      owner: waltju
    taints:
      feaster: "true:NoSchedule"
    tags:
      k8s.io/cluster-autoscaler/node-template/label/owner: "waltju"
      k8s.io/cluster-autoscaler/node-template/taint/feaster: "true:NoSchedule"
      auto-delete: no
    propagateASGTags: true

When I run: $ eksctl create nodegroup -f eksctl-tags.yaml Error: couldn't create node group filter from command line options: loading config file "eksctl-tags.yaml": error unmarshaling JSON: while decoding JSON: json: cannot unmarshal bool into Go struct field ManagedNodeGroup.managedNodeGroups.tags of type string

[Skarlso]

ok, so this extension of eksctl doesn't deliver what we hoped, right?

What do you mean? It's exactly what it's supposed to do.:)

Propagate tags to the ASG. As the value suggests. progapageASGTags.

The thing you are looking for is for unmanaged nodegroups. Managed nodegroups are just tagged with set tags.

Read more here: https://github.com/weaveworks/eksctl/issues/1571

[Skarlso]

But I run into another thing when trying to follow the docs not sure what I am doing wrong:

Please open a separate ticket.

[matti]

@youwalther65 you have the yaml wrong

auto-delete: no

^-- yaml turns no and yes to booleans unless they are inside of "

so

auto-delete: "no"

[youwalther65]

Now I am using:...

managedNodeGroups:
    tags:
      k8s.io/cluster-autoscaler/node-template/label/owner: "waltju"
      k8s.io/cluster-autoscaler/node-template/taint/feaster: "true:NoSchedule"
      auto-delete: "no"
    propagateASGTags: true

and the tags are attched to ASG which is great but with "Tag new instances = No" :

I expected "Tag new instances = Yes" so I can use it for cost related tagging at the EC2 instance level as well! Is there another option or should I raise a feature request for this?

[Skarlso]

The is disabled for a purpose. We didn't want random tags to be attached on to the EC2. If you have a case for it to happen, please open a separate issue and we can triage it. Thanks. :)

[YDKK]

The code propagates TAGS. Not labels or taints. So if you want tags to be propagated to the ASG you need to define tags.

Ok, I understand that automatic tagging for labels and taints works only with unmanaged nodegroups.

However, the documentation for this feature refers to both managed and unmanaged nodegroups, which is very misleading and needs to be corrected.

https://github.com/weaveworks/eksctl/blob/4c1b1421b73e80a6bbe4572602e206f5690ece6e/userdocs/src/usage/autoscaling.md?plain=1#L47-L58

[Skarlso]

@YDKK That's true. Thanks for bringing that up. We'll fix that oversight.

[youwalther65]

The is disabled for a purpose. We didn't want random tags to be attached on to the EC2. If you have a case for it to happen, please open a separate issue and we can triage it. Thanks. :)

I just opened this feature request: https://github.com/weaveworks/eksctl/issues/5443

[Tolsto]

I also just noticed that tags set in the configuration file only get applied to the autoscaling group if you set propagateASGTags to true. However, the schema description states for tags: Applied to the Autoscaling Group, the EKS Nodegroup resource and to the EC2 instances. Therefore, I'd expect tags to get propagated to the ASG even if propagateASGTags is set to false.

[pkit]

@Skarlso it's not propagating. See below: ASG: Launch Template: As you can see tag from the launch template k8s.io/cluster-autoscaler/node-template/label/runtime is not propagated. cluster.yaml:

  - name: ng-2
    instanceType: m6i.12xlarge
    desiredCapacity: 0
    minSize: 0
    maxSize: 4
    availabilityZones: ["us-east-1b"]
    tags:
      k8s.io/cluster-autoscaler/node-template/label/runtime: caspiandb-containerd
    overrideBootstrapCommand: |
      #!/bin/bash
      set -e
      /etc/eks/bootstrap.sh dev --container-runtime containerd --kubelet-extra-args '--node-labels=runtime=caspiandb-containerd'

In our case, it means that AWS cluster-autoscaler is totally broken. As we can see here cluster-autoscaler relies on ASG only: https://github.com/kubernetes/autoscaler/blob/46d7964132ba7c252663b23283a9e17a15dccc09/cluster-autoscaler/cloudprovider/aws/aws_manager.go#L481

[Skarlso]

The above config doesn't have propagateAsgTags set to true?

[pkit]

It was the default...

[Skarlso]

Sorry, not following. propagateASGTags is false by default.

[pkit]

It was true by default not a long time ago.

[Skarlso]

I don't recall. :) You might be right, I worked on a lot of other things. However, right now, it's false by default. :) I can't see any changes to the doc saying otherwise. But I might be wrong. :))

[Himangini]

We will investigate this issue and also check the default setting for propagateAsgTags, and if there are documentation changes required we will add those accordingly.

[nikimanoledaki]

I just tested all these scenarios with eksctl 0.106.0. Here are the results:

Unmanaged Nodegroups

1. tags set, propagateASGTags not set The tags were propagated as ASG tags.

2. tags and propagateASGTags set The tags were propagated as ASG tags.

3. labels and taints set, tags not set, and propagateASGTags set The labels and taints were propagated as ASG tags.

Managed Nodegroups

4. tags set, propagateASGTags not set The tags were NOT propagated as ASG tags.

5. tags and propagateASGTags set The tags were propagated as ASG tags.

6. labels and taints set, tags not set, and propagateASGTags set The labels and taints were NOT propagated as ASG tags.

Conclusions

• propagateASGTags propagates tags, labels, and taints for unmanaged nodegroups, and only tags for managed nodegroups.
• tags alone propagate to the ASG by default for unmanaged nodegroups, NOT for managed nodegroups.

Apologies for any confusion. We'll be updating the documentation to reflect the current code as clearly as possible. I'm looking into why we have not added support for this for managed nodegroups as well given the previous requests.

[nikimanoledaki]

Added a PR to update the docs to reflect the current state of eksctl. Hopefully this makes things clearer regarding the difference between managed vs unmanaged nodegroups. It also clears up the behaviour of tags, which are propagated by default for unmanaged nodegroups but NOT for managed nodegroups. Thanks @YDKK for pointing this out and everyone for your feedback!

Lastly, I opened an issue so that propagateASGTags can propagate labels+taints as ASG tags for managed nodegroups as well once and for all 😄 🔜