instanceSelector support for allow/deny list

[kr3cj]

We are starting to use the instanceSelector feature in some our EKS configs. It greatly simplifies our configuration and is easier to understand. Thank you for this feature.

However, in order to use it everywhere, we would need to be able to specify some kind of allow/deny list for specific instanceTypes.

The reason we need this is because we've performed profiling of different instance classes with our workloads and found that certain ones aren't something we want to adopt in production (t3 specifically). They're good for staging/dev environments but for production, we try to stick with other instance classes.

We noticed that the underlying ec2-instance-selector already supports the following arguments:

      --allow-list string                 List of allowed instance types to select from w/ regex syntax (Example: m[3-5]\.*)
      --deny-list string                 List of instance types which should be excluded w/ regex syntax (Example: m[1-2]\.*)

It would be great if those could be exposed in the eksctl config schema. Something like:

managedNodeGroups:
  - name: my-node-group
    instanceSelector:
      vCPUs: 4
      memory: "16"
      allow: "[cm]5\.*"

Or

managedNodeGroups:
  - name: my-node-group
    instanceSelector:
      vCPUs: 4
      memory: "16"
      deny: "t\.*"

This came out of the discussion on https://github.com/weaveworks/eksctl/issues/3531#issuecomment-814225408

[cPu1]

This is a useful feature, but we're wary of adding more instance selector options to eksctl to avoid bloating it with features only few are using. We're going to discuss this with the EKS team to check if other users would benefit from this feature or not.

[bhops]

This is a useful feature, but we're wary of adding more instance selector options to eksctl to avoid bloating it with features only few are using. We're going to discuss this with the EKS team to check if other users would benefit from this feature or not.

I would argue that part of the reason it may not be very widely used could be because of how limited the included options are. For my use case, the inability to filter the list of returned instances is a non-starter for using the instanceSelector option entirely as it brings in very old generation as well as burstable instances which we do not want running in production.

[akestner]

I'd be curious to hear if an integration with the new EC2 Attribute Based Selection feature would satisfy your needs. For example, you could exclude the burstable, t2/3/4 instance family using ABS's ExcludedInstanceTypes override while remaining flexible about the instances that get provisioned for your cluster.

[bhops]

@akestner For my use case this looks like it would satisfy our requirements. This actually looks more flexible, since it seems like we can do things like the following.

ArchitectureTypes:  # [REQUIRED] 
- x86_64
VirtualizationTypes: # [REQUIRED] 
- hvm
InstanceRequirements: # [REQUIRED] 
  VCpuCount:
    Min: 4
    Max: 6
  MemoryMiB:
    Min: 2048
  InstanceGenerations:
    - current
  BurstablePerformance: excluded

[kr3cj]

@akestner , yes an integration with new EC2 Attribute Based Selection feature would likely meet our needs as well.

This also seems related, or at least a good time, to also refer to https://github.com/aws/containers-roadmap/issues/1297 as the combination of both of these ideas would greatly reduce our managedNodeGroup design and configs. Currently we run a mix of spot and ondemand in our production clusters. But these must be separate node groups from each other.

[kr3cj]

I'd be curious to hear if an integration with the new EC2 Attribute Based Selection feature would satisfy your needs. For example, you could exclude the burstable, t2/3/4 instance family using ABS's ExcludedInstanceTypes override while remaining flexible about the instances that get provisioned for your cluster.

Would you prefer a new issue be opened for that integration, @akestner ?

[matti]

Currently c5.metal instances just crash - there is no way to prevent this without deny support when requesting 96cores which gives metal and non-metal instances https://github.com/aws/amazon-vpc-cni-k8s/issues/1929#issuecomment-1067993183