containers-roadmap
containers-roadmap copied to clipboard
aws
[EKS] [request]: Managed Nodes scale to 0
Currently, managed node groups has a required minimum of 1 node in a node group. This request is to update behavior to support node groups of size 0, to unlock batch and ML use cases.
https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#how-can-i-scale-a-node-group-to-0
This feature would be great for me. I'm looking to run GitLab workers on my EKS cluster to run ML training workloads. Typically, these jobs only run for a couple of hours a day (on big instances) so being able to scale down would make thing much more cost effective for us.
Any ideas when this feature might land?
@mathewpower you might want to use a vanilla autoscaling group instead of EKS managed.
Pretty much this issue makes EKS managed nodes a nonstarter for any ML projects due to one node in each group always being on
I guess that node taints will have to be managed like node labels already are in order for the necessary node template to be set: https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md#scaling-a-node-group-to-0.
Hey @yann-soubeyrand that is correct. Looking for some feedback on that, would you want all labels and taints to automatically propagate to the ASG in the required format for scale to 0, or have selective control over which ones propagate?
@mikestef9 If AWS has enough information to propagate the labels/taints to the ASG, then I think it'd be preferable to have it "just work" as much as possible.
There will still be scenarios where manual intervention will be needed by the consumer I think such as setting region/AZ labels for single AZ nodegroups so that cluster-autoscaler can make intelligent decisions if a specific AZ is needed, however we should probably try to minimize that work as much as possible.
@mikestef9 in my understanding, all the labels and taints should be propagated to the ASG in the k8s.io/cluster-autoscaler/node-template/[label|taint]/<key> format since the cluster autoscaler takes its decisions based on it. If some taints or labels are missing, this could mislead the cluster autoscaler. Also, I'm not aware of any good reason not to propagate certain labels or taints.
A feature which could be useful though, is to be able to disable cluster autoscaler for specific node groups (that is, not setting k8s.io/cluster-autoscaler/enabled tag on these node groups).
@dcherman isn't the AZ case already managed by cluster autoscaler without specifying label templates?
@yann-soubeyrand I think you're right! Just read through the cluster-autoscaler code, and it looks like it discovers what AZs the ASG creates nodes in from the ASG itself; I always thought it had discovered those from the nodes initially created by the ASG.
In that case, we can disregard my earlier comment.
I would like to be able to forcibly scale a managed node group to 0 via the CLI, by setting something like desired or maximum number of nodes to 0. Ignoring things like pod disruption budgets, etc.
I would like this in order for developers to have their own clusters which get scaled to 0 outside of working hours. I would like to use a simple cron to force clusters to size 0 at night, then give them 1 node in the morning and let cluster-autoscaler scale them back up.
Hi All is this feature already for AWS EKS? From following documentation it appears EKS supports it - From CA 0.6 for GCE/GKE and CA 0.6.1 for AWS, it is possible to scale a node group to 0 https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#how-can-i-scale-a-node-group-to-0 Can someone please confirm?
Hi All is this feature already for AWS EKS? From following documentation it appears EKS supports it - From CA 0.6 for GCE/GKE and CA 0.6.1 for AWS, it is possible to scale a node group to 0 https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#how-can-i-scale-a-node-group-to-0 Can someone please confirm?
@sibendu it's not supported with managed node groups yet (this is the object of this issue) but you can achieve it with non managed node groups (following the documentation you linked).
Would be great to have this, we make use of cluster autoscaling in order to demand GPU nodes on GKE and scale down when there are no requests. Having one node idle is definitely not cost effective for us if we want to use managed nodes on EKS
Putting use cases aside (although I have many), autoscaling groups already support min, max & desired size being 0. A node group is ultimately just an autoscaling group (and therefore already supports size 0). You can go into the AWS web console, find the ASG created for a node group and set the size to 0 and it's fine therefore it doesn't make sense that node groups are not supporting a zero size. As a loyal AWS customer it's frustrating to see things like this - there appears to be no good technical reason for preventing a size of zero but forcing customers to have a least 1 instance makes AWS more £££. Hmmm... was the decision to prevent a zero size about making it better for the customer or is Jeff a bit short of cash?
@antonosmond there are good technical reasons why you cannot scale from 0 with the actual configuration: for the autoscaler to be able to scale from 0, one have to put tags on the ASG indicating labels and taints the nodes will have. These tags are missing as of now. This is the purpose of this issue.
@yann-soubeyrand The cluster autoscaler is just one use case but this issue shouldn't relate specifically to the cluster autoscaler. The issue should be that you can't set a size of zero and regardless of use case or whether or not you run the cluster autoscaler, you should be able to set a size of zero as this is supported in autoscaling groups.
In addition to the use cases above, other use cases for 0 size include:
- PoCs and testing (I may want 0 nodes so I can test my config without incurring instance charges)
- having different node groups for different instance types where I don't necessarily need all instance types running at all times
- cost saving e.g. scaling to zero overnight / at weekends
@antonosmond if you're not using cluster autoscaler, you're scaling the ASG manually, right? What prevents you from setting a min and desired count to 0? It seems to work as intended.
@yann-soubeyrand I got to this issue from here. It's nothing to do with the cluster autoscaler, I simply want to create a node group with an initial size of 0. I have some terraform to create a node group but if I set the size to 0 it fails because the AWS API behind the resource creation validates that size is greater than zero. Update - and yes I can create a node group with a size of 1 and then manually scale it zero but I shouldn't need to. The API should allow me to create a node group with a zero size.
The API should allow me to create a node group with a zero size.
I think we all agree with this ;-)
Not having this makes it exceptionally hard migrating from one Node Group to another (we are in the process of moving to our own launch templates) without fearing breaking everything without a good rollback procedure.
I agree, this would be a great feature. Having to drain + change ASG desiredInstanceCount is tedious. I have an infrequently accessed applicaiton running on EKS that I spin up when needed, but don't need it to sit idle at 1 instnace even when not being used. Any update on timeline?
Looking to see if there's any update here?
I believe this is preventing me from having multiple different instance types across multiple different node groups. If I want to have a node group for each size of m5s, now I have to have at least 1 running for each as well even if it's unlikely that I need the 2xl or 4xl.
Adding some noise here. Spot instances was one of our hurdles (thanks for delivering!), but we are holding off on moving to managed node groups until we can be assured we won't have large, idle nodes for the sporadic bigger workloads. Any updates here would be helpful.
Yep, this is a bummer and certainly makes migrating from ASGs to managed node groups much less appealing. +1 for this feature.
Update - and yes I can create a node group with a size of 1 and then manually scale it zero but I shouldn't need to. The API should allow me to create a node group with a zero size.
When you say manually scale it to zero, do you mean literally change the desired value of the ASG after the fact? Is that permanent - at least until you re-deploy the infra? @antonosmond
@calebschoepp It seems to be so far, including after running upgrades on the nodegroups. I actually do this using a local-exec provisioner in Terraform after creation of the nodegroup for the ones that I want scale to 0 on.
We make heavy use of GPU nodes in EKS with Jupyter Notebooks that will autoscale based on requests and prune notebooks after inactivity. It makes it impossible for us to migrate to Managed Nodes as GPU instances are so expensive and we need one always on. Hoping this gets released sooner than later 👍
I have unfrequent bursts of heavy workloads requiring many resources. It doesn't make sense to keep a machine running at all times. Please make this happen!
Almost a year since this issue has been brought up. Any update or timeline on when we might have it?
Thank you!
I want to downsize dev node groups when no work is actively being done. Please add this feature!
