Spot VM Support

[vlerenc]

Stories

• As user/operator I want to use AWS spot, Azure Low-Priotity and/or GCP Preemptible VM instances, so that my landscape runs at lower costs.

Motivation

Money, sure, but also some form of chaos monkey that should help train the application developers that all resources will eventually fail.

Acceptance Criteria

• [ ] Machine classes can be specified as having one of these additional "flavors":
  • [ ] https://aws.amazon.com/en/ec2/spot/ (requires some field spot bid/price, most likely)
  • [ ] https://cloud.google.com/compute/docs/instances/preemptible (seems to be no bidding involved here)
  • [ ] https://azure.microsoft.com/en-gb/blog/announcing-public-preview-of-azure-batch-low-priority-vms/ (seems not to be general purpose?)
• [ ] Machines are either (at least) labelled or maybe even tainted so that only tolerating pods (with strict replication onto multiple machines) are scheduled onto them

Remarks

Looks like Bosh had the same idea (well, everybody can if they have cattle VMs).

Enhancement/Implementation Proposal (optional)

Ideally, link to EP, e.g. a GEP in Gardener (https://github.com/gardener/gardener/tree/master/docs/proposals), alternatively prose here.

Challenges

• supporting autoscaling for such node-grps. Known issues are:
  • https://github.com/kubernetes/autoscaler/issues/4958
  • https://github.com/kubernetes/autoscaler/issues/1133#issuecomment-484117501
• #352

[hardikdr]

GKE enabled support for Pre-emptible VMs: https://cloud.google.com/kubernetes-engine/docs/concepts/preemptible-vm

[vlerenc]

Yes, I saw that quite some time ago. That's why I said in one of our syncs, we won't be the first anymore. It really does make a lot of sense, too. On the other hand, our priorities are right. We know we like to have it eventually, but we can't do everything at the same time.

[vlerenc]

Funny, today I even saw this (thanks @afritzler): https://cloudplatform.googleblog.com/2018/06/Cloud-TPU-now-offers-preemptible-pricing-and-global-availability.html

Maybe we can beat GKE with preemptible TPUs in Kubernetes clusters then? ;-) Just kidding, but TPU support is definitely also interesting and somehow different from how AWS handles GPU support (that already works, because MCM doesn't care, but TPUs must be assigned, @afritzler and @rfranzke told me a couple of days ago).

[vlerenc]

Once we have the time to work on this one (GKE and others support that, too - just saw it with Banzai as well), we might leverage this here: org:banzaicloud repo:spot-price-exporter.

[samedguener]

@vlerenc Are there any updates regarding node pools with hotspot instances? We are looking forward for this!

Best, Samed

[vlerenc]

No, no update. So far, nobody even contacted us with the concrete need. You are the first. Most workload can't cope with that kind of infrastructure. Can you elaborate about your use case a bit?

cc @hardikdr @prashanth26 @amshuman-kr @juergenschneider

[samedguener]

We are planning to have node pools with hotspot instances

• to run in future batch jobs such as the training of machine learning model (long running batch jobs) on hotspot instances and reschedule them during unavailability. This will allow us to reduce our TCOs on the long-term.
• reduce cost in our development costs by allowing to scheduling pods during PRs tests to such nodepools.

Best, Samed

[damondd]

Have similar scenario. We need to dynamically create a pool of preemptible nodes for running divided cpu/gpu computing tasks (each task may last from 10 min to hours e.g.), expect to have some management like applying new node once any recalled.

[prashanth26]

This is currently enabled on AWS with - https://github.com/gardener/machine-controller-manager/pull/481. However, integration with gardener is yet to be done.

• [x] Support spot instances for AWS
• [ ] Integration with gardener

[sebarys]

Hi, We're having similar scenario: we would like to run stage/test cluster using AWS Spot fleet to reduce costs. Is there any pland to deliver it in near future? Thanks in advance for response :)

[amshuman-kr]

It is certainly part of the roadmap, but we have a couple of other things (like improving draining of pods across cloud providers) in the plan before we can get to it.

[jscarney]

Update on ability to support 'spot' instances across Azure and GCP? Will be useful towards cost savings.

[vlerenc]

There were quite some updates: E.g. AWS, Azure, and GCP now all support spot instances with dynamic prices (Azure and GCP deprecated their old models in favour of the new ones that are all called spot VMs). GCP doesn't support a threshold though, which is less than optimal (you can always look up the price though and act accordingly). Grace periods vary (AWS 120s, Azure and GCP 30s), but all notify and we could use that for immediate drain.

I also looked up auto-scaling groups: Now they all support multiple zones, but only AWS and Azure support mixing on-demand and spot instances. AWS' feature seems strange though, because different than Azure and GCP, the spot price may go beyond even the on-demand price. When the user sets a limit, e.g. at the regular on-demand price, AWS won't add capacity and you are left with the on-demand baseline, but Azure fulfils the request, capped at the on-demand price, so you get your machines still. That's at least how I understood the docs.

Rebalancing is another open point, e.g. never, always, grace_period and/or cost_gap maybe?