Max Pods Calculator Kaizen

[o6uoq]

AWS EKS has the notion of calculating ENIs per Instance, which once calculated, provide you with the number of Pods which can be run per Instance.

Mapping is calculated from AWS EC2 API using the following formula:

• First IP on each ENI is not used for pods
• +2 for the pods that use host-networking (AWS CNI and kube-proxy)

Number (#) of ENI * (# of IPv4 per ENI - 1) + 2

👉🏼 https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#AvailableIpPerENI

max-pods-calculator.sh helps as a Max Pod Calculator, however, it calculates on a binary outcome e.g. two outcomes: (1) MAX_POD_CEILING_FOR_LOW_CPU and (2) MAX_POD_CEILING_FOR_HIGH_CPU, which is based on if the CPU_COUNT is higher or lower than 30.

What would you like to be added:

• ENI and IP calculation takes into account when using Pod Security Groups
• Example use cases and support for Pod Security Group users/customers
• Max Pods can be overridden explicitly

Why is this needed:

The calculation does not take into account two things:

1. ENI / IP calculation when using Pod Security Groups
2. Ability to override MAX_POD_* given a explicit setting (existing PR somewhat related)

It is needed so that all AWS EKS customers, whether they use Pod Security Groups or not, have feature parity in the use of this script to provide the most accurate output of ENI, IP and CPU allocation based on the subjective use of AWS.

[stevehipwell]

@o6uoq you can manually overwrite max pods by setting the env variable USE_MAX_PODS to false and passing in --max-pods, set to the desired value, to the kubelet extra args.

You might want to take a look at the official Kubernetes large clusters guide which puts a limit of 110 pods per node.

Remember that when changing the max pods you also should pass in --kube-reserved to the kubelet extra args as otherwise it will still use the ENI limits for the default instance pod count (see https://github.com/awslabs/amazon-eks-ami/issues/782 for more details).

[endersonmaia]

ou can manually overwrite max pods by setting the env variable USE_MAX_PODS to false and passing in --max-pods, set to the desired value, to the kubelet extra args.

@stevehipwell is that only doable via launch template for the managed nodeGroup, or is there any other way ?

I'm currently using CDK to define the nodegroup for my EKS cluster, but not defining any launchTemplate yet.

[stevehipwell]

@endersonmaia you'd need to use a launch template to do this; for MNGs you'd need also to patch bootstrap.sh in your user data (like I've automated in https://github.com/awslabs/amazon-eks-ami/pull/855) to be able to achieve this.