Cluster Autoscaler doesn't scale ASG from 0 unless manually scaled first (and afterward works as expected).

[ZTGallagher]

If we have at least once scaled up the ASG while the cluster autoscaler is installed, it can from then one scale the deployment to and from 0.

But if the ASG has never once been scaled up, it won't work.

Here is the IAM role permissions set for the cluster autoscaler.

{
    "Statement": [
        {
            "Action": [
                "ec2:DescribeLaunchTemplateVersions",
                "ec2:DescribeInstanceTypes",
                "autoscaling:DescribeTags",
                "autoscaling:DescribeLaunchConfigurations",
                "autoscaling:DescribeAutoScalingInstances",
                "autoscaling:DescribeAutoScalingGroups"
            ],
            "Effect": "Allow",
            "Resource": "*",
            "Sid": "eksAutoscalingAll"
        },
        {
            "Action": [
                "eks:DescribeNodegroup",
                "ec2:DescribeInstanceTypes",
                "autoscaling:TerminateInstanceInAutoScalingGroup",
                "autoscaling:SetDesiredCapacity"
            ],
            "Condition": {
                "StringEquals": {
                    "autoscaling:ResourceTag/k8s.io/cluster-autoscaler/enabled": "true",
                    "autoscaling:ResourceTag/kubernetes.io/cluster/use1-cluster": "owned"
                }
            },
            "Effect": "Allow",
            "Resource": "*",
            "Sid": "eksAutoscalingOwn"
        }
    ],
    "Version": "2012-10-17"
}

Tags on the ASG.

"k8s.io/cluster-autoscaler/node-template/label/nvidia.com/gpu": "true"
"k8s.io/cluster-autoscaler/node-template/resources/nvidia.com/gpu": "1"

The tags match the label and gpu the deployment looks for.

I don't know how Cluster Autoscaler caches which ASG applies to which labels. But it works after at least one manual scale-up. That's not really a viable solution though. The whole deployment would be essentially hands-off without this issue.

[dev-rowbot]

@ZTGallagher - have a look at my comment on #4998 - you are probably experiencing the same issue.

[WillerWasTaken]

We also got these errors when the size of the node group is initially 0

I0907 15:13:41.616856       1 scale_up.go:300] Pod test-9b7646c8-8w5w6 can't be scheduled on eks-test-20220907151106983600000009-6cc18c40-9fd2-92f8-2fb4-7cb775be3182, predicate checking error: node(s) didn't match Pod's node affinity/selector; predicateName=NodeAffinity; reasons: node(s) didn't match Pod's node affinity/selector; debugInfo=

We're using eks_managed_node_groups and we (finally) got it working by setting the scaling options like this

scaling_config {
  desired_size = 1
  max_size     = 5
  min_size     = 0
}

The important bit is that we have the initial desired_size at 1 instead of 0. It allows the side effect to occur to make the node group eligible by the autoscaler. After this it all works fine and the autoscaler can properly scale in at 0 and back again at 1 whenever it's necessary.

It's more or less like manually setting the desired size to 1 after creation. Not the greatest workaround

[BillKalaitzisXentral]

We're also having the exact same issue. The autoscaler is unable to scale a nodegroup from 0, unless we've manually scaled it up previously.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.