Awareness of node hourly billing cycle

[jonathon2nd]

Which component are you using?:

Cluster autoscaler.

---

Is your feature request designed to solve a problem? If so describe the problem this feature should solve.:

When deploying a workload that auto-scales or when hosting a workload which creates jobs as needed, it can be desired to host this in public cloud with managed k8s and node-pool autoscaling.

However, if nodes are added to the node pool and then 20 or so minutes later there is no more work, the autoscaler will then remove these nodes. These nodes may be billed hourly, and they are being decommissioned too soon. New work may come in and they can be of further use, without having to create new nodes at additional cost.

---

Describe the solution you'd like.: I would like the ability to set as a parameter in the autoscaler that the node-pool nodes are being billed hourly.

This would result in nodes being kept for longer, until it is closer to the end of their hour. Then if they are still candidates for downscaling, only then are they decommissioned.

If by doing this the node rolls over to the next hour, then this process would start over and that node would be retained until close to the end of it's billing cycle

---

Describe any alternative solutions you've considered.:

I have been brainstorming a script that would take the metrics of the workload which deploys jobs. These jobs run on the autoscaling node-pool. Deriving the number of nodes based off those metrics is not difficult. This script would then set desiredNodes, and the autoscaler would take care of the rest.

I can't see a way to do that, and keep nodes from being deleted too soon.

I can not see a way to retain nodes for closer to the end of their billing cycle, without disabling the autoscaler entirely and managing all the logic. I have gone down the road of (on-prem and public cloud) complete management of nodes and doing all the logic for scaling up and scaling down (evaluate nodes, cordon, recheck things, drain, delete node). I would ideally like to avoid this.

EDIT: I just come up with another idea, this one may actually work. Instead of having a script running scaling up and down the node-pool via setting desiredNodes, have it add and remove taints on nodes based of their age.

By using this https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#how-can-i-prevent-cluster-autoscaler-from-scaling-down-a-particular-node I can write a script that constantly runs, and add/removes this taint based off it's age.

The autoscaler should be able to work with this.

If it goes to scale down and we have the node tainted based on its age, it should leave it alone. Then when it gets close to the end of the billing cycle, and it is still not in use and if the autoscaler still wants to remove it then the taint would be removed and the autoscaler will decom the node.

---

Additional context.:

I had found this issue: https://github.com/ovh/public-cloud-roadmap/issues/223 Because we are using OVH managed k8s for this application. However I am thinking that it might need to be addressed here, as I see from their doc that it is using kubernetes autoscaler.

see here: https://docs.ovh.com/us/en/kubernetes/configuring-cluster-autoscaler/

You can get more information on those parameters on the Kubernetes autoscaler documentation.

[nantoniazzi]

Thanks for your report and mention, I see that we have the same issue :) . Your workaround based on taints is interresting, I did not think about this option.

I will try to create a cronjob associated with the proper serviceaccount to see if it's doable without too many problems.

We also have monthly/hourly billing issues: https://github.com/ovh/public-cloud-roadmap/issues/67#issuecomment-1152435819

Maybe a solution based on the OVH billing system (similar to AWS saving plans) could solve this problem too.

[jonathon2nd]

So far the node-annotator chart I have written is working well. I will be fine-tuning it for a bit as I see how it behaves in prod, but so far I am happy with it. If it continue to work well and I get it polished enough I may make it public.

[jonathon2nd]

I see that the creation time of the machine in OVH PC is already known https://github.com/kubernetes/autoscaler/blob/8037b0f9f5485e7b0d7c8b93cdabe3251aaf4fcb/cluster-autoscaler/cloudprovider/ovhcloud/sdk/node.go#L37

I am wondering if it is possible to add this as a label onto the nodes?

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.