Ability to set VPC CNI configuration variables per instance type

[olileach]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request What do you want us to build?

I would like to be able to set VPC CNI configuration variables https://github.com/aws/amazon-vpc-cni-k8s#cni-configuration-variables per instance type using userdata (or another mechanism). When the VPC CNI pod starts on the host, I would like it to check for any variables I've set on the node and use those values instead of the global values set via the VPC CNI.

For example, MINIMUM_IP_TARGET and WARM_IP_TARGET configuration for the VPC CNI is a global setting and is applied to all instances deployed via kubernetes, regardless of whether these nodes are deployed via mechanisms like the Kube API, CloudFormation, EKSCTL, Cluster Autoscaler or Karpenter etc.

I would like to have the ability to set MINIMUM_IP_TARGET and WARM_IP_TARGET for either specific instance types deployed by CAS or Karpenter, for example.

Which service(s) is this request for? This could be EKS, Fargate

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?

I am trying to solve the over allocation of IPs when using MINIMUM_IP_TARGET for all instance types. Say I have a batch workload which scale up, I want use 60 IPs for large instance types and a MINIMUM_IP_TARGET. But for smaller nodes in other node groups that manage other types of pods, will use this value and consume unnecessary IPs.

By setting these values on the node via userdata, I can control different instance types MINIMUM_IP_TARGET variable value. Different instance types can run different numbers of pods, therefore the MINIMUM_IP_TARGET can and should be different. Smaller instance types could have a smaller MINIMUM_IP_TARGET value whereas larger instance types could have a larger MINIMUM_IP_TARGET value.

It is currently hard to set these values per instance type using custom code as these are global settings applied via teh VPC CNI.

Are you currently working around this issue?

We're not but we are experimenting with init containers to see I we can set values ourselves

Additional context Anything else we should know?

I logged a similar feature request here but was told to log it within the containers roadmap.

Attachments If you think you might have additional information that you'd like to include via an attachment, please do - we'll take a look. (Remember to remove any personally-identifiable information.)