Enhance the AWSMachineTemplate to support multiple instance types, enable mixed instances in a MachineDeployment

[cnmcavoy]

/kind feature

Describe the solution you'd like Enhance the AWSMachineTemplate to support multiple instance types, specifically an ordered of instances and spot market options to use when creating the ec2 instance to back an AWSMachine, in order to handle provisioning failures when the desired instance type is unavailable. The AWSMachine should provision the ec2 instance with the type and spot options of the first item of the list, and only walk the list and use the other configurations if there is insufficient capacity to provision.

User stories:

• As a user, I want to be able to create a MachineDeployment which can be satisfied with multiple instance types of the same capacity but different families. For example, I want to create a node group for m6a.32xlarge or m6i.32xlarge instances, and I want to try creating m6a instances if they are available, but if they are unavailable, use m6i's.

• Also, as a user I want to be able to create a MachineDeployment which can use spot instances, but fall back to on-demand instances when the spot market is empty. For example, I want to create a node group for t3.large, but specify spot options to get a better price, and then fall back to on-demand t3.large's when the spot market is empty.

Anything else you would like to add: This would enable mixed instance capacities, e.g a m6a.2xlarge and m6a.4xlarge in the same MachineDeployment, however this kind of configuration is unsupported by the cluster autoscaler, so we may need to improve documentation to state that mixed capacities are unsupported in CAPI as well.

Environment:

• Cluster-api-provider-aws version: v2.0.2
• Kubernetes version: (use kubectl version): v1.24
• OS (e.g. from /etc/os-release): Ubuntu 20.04