containers-roadmap
containers-roadmap copied to clipboard
aws
[EKS] [request]: Nodegroup should support tagging ASGs
Community Note
- Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
- Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment
Tell us about your request It would be great if we could pass tags to the underlying ASGs (and tell the ASGs to propagate tags) that are created from the managed node groups for EKS so that the underlying instances/volumes are tagged appropriately.
Which service(s) is this request for? EKS
Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? Currently, managed node groups for EKS do not support propagating tags to ASGs (and therefore the instances) created by the node group. This leads to EC2 instances that are not tagged according to our requirements for tracking cost, and resource ownership.
Passing the managed node tags to launch templates "Instance tags" will automatically apply to both EC2 and its volumes. If there are some challenges to do that, creating a separate "Custom Tags" section in the EKS managed node configuration page will also be helpful.
Workaround to add custom tags to WorkerNodes using EKS managed NodeGroup :
-
Create a managed worker Node group in EKS console. (Set Minimum & desired count as 1)
-
EKS creates an ASG in the background. You will find ASG information for EKS NodeGroup details in EKS console. Select ASG associated with managed worker NodeGroup > tags > Add your custom tags for EC2.
Note: Make sure to checkbox "Tag New Instances" while creating new tags. -
Terminate the newly launched Ec2 without tags.
-
Scale up ManagedNodeGroup as per requirement.
-
After completing above steps, EKS managed Node groups will tag new EC2 Instance with custom tags.
This is crucial feature that is missing, and is the only reason our department is not moving from manual ASGs to node groups.
Yes, this is a very important change. We also cannot use this because of the need for tags. Bad practice to use semi-automatic infrastructure as code.
any updates ? can you open source node groups.. so, community can contribute ?
I don't think it's a duplicate. This one is for an API feature to add tags to the ASG created by the API, and also be able to set the flag on the ASG that propagates tags outwards: so it's only an API change to implement the same thing down manually in the workaround above.
#374 is for the EKS Cluster object itself to support propagating tags down, in the way ASGs already do. I imagine #374 would partially work by propagating tags to ASGs, and then turning on ASG tag propagation, rather than duplicating the behaviour.
Team: Having this functionality available will enable customers to use Cluster Autoscaler's capacity autodiscovery feature instead of forcing them to maintain manual capacity mappings on the command line.
The documentation there isn't super clear (see https://github.com/kubernetes/autoscaler/pull/3198 for documentation updates), but advertising capacity resources to Cluster Autoscaler via ASG tags will make the use of multiple heterogeneous Auto Scaling Groups much easier for customers.
Team: Having this functionality available will enable customers to use Cluster Autoscaler's capacity autodiscovery feature instead of forcing them to maintain manual capacity mappings on the command line.
The documentation there isn't super clear (see kubernetes/autoscaler#3198 for documentation updates), but advertising capacity resources to Cluster Autoscaler via ASG tags will make the use of multiple heterogeneous Auto Scaling Groups much easier for customers.
@otterley While Managed Nodegroup doesn't support customer provided tags for ASGs today, we do add the necessary tags for CAS auto discovery to the ASG i.e. k8s.io/cluster-autoscaler/enabled and k8s.io/cluster-autoscaler/<CLUSTER NAME>.
@rtripat Understood. Perhaps I wasn't clear, but I was specifically referring to the ability to autodiscover specific capacity dimensions of an ASG such as cpu, memory, ephemeral storage, GPU, etc.
Until this feature is ready, I've had success with creating a cloudwatch rule based upon EC2 "pending" status, invoking a lambda that checks the instance_id passed in through the event, checks the instance_id to see if it's part of a managed node cluster, then adds the appropriate tags. I'm doing this all through Terraform with the spin up of the eks cluster.
Obviously would be much easier with a tags option! 😛
It could be great to be able to tag the launch templates too with the option to propagate these tags to instances and volumes or not.
Is there some kind of best practice on tagging ASG vs tagging LT? It seems to me that tagging LT offers more flexibility (like the ability to tag the volumes).
https://docs.aws.amazon.com/autoscaling/ec2/userguide/autoscaling-tagging.html touches upon the overlap in tag propagation between ASGs and Launch Templates.
https://docs.aws.amazon.com/autoscaling/ec2/userguide/autoscaling-tagging.html touches upon the overlap in tag propagation between ASGs and Launch Templates.
That's precisely the documentation page I had in mind when asking about best practices ;-) This page explains the overlap but there are no clear pros and cons of the two tagging approaches. But it seems to me that LT offers more flexibility and that ASG tags should be used only when necessary (like for the cluster autoscaler discovery tags).
There's a related discussion about tagging ASGs and LTs for non-managed Nodegroups at https://github.com/weaveworks/eksctl/issues/1603. My understanding from there is that tagging LTs and enabling propagation would be sufficient, but there might be use-cases where the ASG needs to have the tag too, but it wouldn't then needed to also support propagation.
The difference observed in that ticket is that the ASG propagation applies the tags after launch, while LT propagation applies the tags as part of the launch.
Yes, I create my non-managed node groups using Terraform and put the tags on the LT with propagation to instances and volumes. The only tags I needed to put on ASG are the cluster autoscaler related tags. But propagation is not needed for these tags.
need this feature too, will impact calculate costs if I add the tags manually later in ASG.
We have EKS deployed as a new part of our stacks in prod through preprod, stage and dev (alongside a very large ECS deployment in each environment). It is very annoying that the instances are not tagged for cost allocation.
I would also like to see custom names or name prefixes for the autoscaling groups. The auto-generated uuid naming really slows down management of larger clusters.
With managed node groups support for launch templates, you can now add tags to the EC2 instances created as part of your node groups. See EKS docs for details.
I will leave this issue open for a little while, as I want to get some more feedback. The issue as originally opened asks for tags on ASGs, but I suspect most of you ultimately care about tags on EC2 instances, not the ASGs. Please leave any comments if you still have a need for tags on the ASGs themselves. Our vision is we handle any of these ASG tags for you, for example when we implement scale to 0 #724, we'll automatically add the required tags to the ASG.
I will leave this issue open for a little while, as I want to get some more feedback. The issue as originally opened asks for tags on ASGs, but I suspect most of you ultimately care about tags on EC2 instances, not the ASGs.
As the original issue creator, I can confirm that being able to tag the underlying EC2 instances was indeed the intent of the original ask. Though others may have had other reasons for wanting ASGs to be tagged.
Thank you to the EKS team for implementing this!
Tags on the ASG are crucial if the ASG scales to zero. The cluster autoscaler for example will use the ASG tags if they exist. Without a way to propagate tags to the ASGs, we either have to run with unnecessary hosts or we have to bootstrap the ASGs directly.
@dindurthy As I mentioned above, "Our vision is we handle any of these ASG tags for you, for example when we implement scale to 0 #724, we'll automatically add the required tags to the ASG."
@mikestef9 While it is an admirable goal to handle the ASG tags automatically, it seems unlikely you will be able to do it quickly or easily. There are tags for node labels, node taints, and node resources, and it is unlikely EKS will be aware of all of them because of the various ways they can be created. At the moment it appears I cannot even get tags to propagate from the launch template to elastic GPUs (it seems EKS makes a copy of the launch template rather than use it directly, and the copy disables the "tag elastic graphics" setting), which makes me wary of trusting automatic behavior. I would rather you implement direct ASG tagging (or at least copying Launch Template tags to the ASG) first and see about automation later.
Did the tags used to work? I thought they did but all my nodeGroups now no longer have the tags specified in the eksctl configuration. Need to get these tags back in as they are used for Cost reporting.
Docs says the feature is there now but it does not work. Is it a tf 0.13.x feature as I am still on 0.12.x (don't want to move to 0.13 yet)? Would be nice the EC2 worker instances had a meaningful name, rather than just a hyphen.
tags = {
"Name" = "eks-${var.cluster-name}-1"
}
I really want for tags on ASGs.
Because I attach target group to my ASG ( we don't use type: loadbalancer / use nodeport) . Since we have multiple ASGs for different purposes, we need to be able to identify them and attach the Target Group.
