[EKS] [request]: Increase branch ENI Density

[sachin-net]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request With the recently launched feature of using "Security group for Pods", it leverages Branch ENI and the number of the Pods that can be spawned is only around 1/3 of the original Density.

If we take example of c5.large / m5.large it supports up-to 29 Pods per host, however with Security groups assigned to Pods, only 9 Pods can be launched.

Which service(s) is this request for? EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?

Currently we are using Calico Network policies to restrict egress communication to RDS. With this new feature of assigning "security group for pod", it would be more efficient to restrict egress communication to RDS.

We have special use-case where one user is mapped to one pod and we run lot of very small backend Pods and these backend pods would further communicate with RDS. Due to very high number of small pods (each pod would use around 75 to 100m CPU requests allocated), Pod density is very critical factor.

Are you currently working around this issue? By not using "Security group for Pod" feature and continue using Calico network policies to restrict egress communication to RDS.

Additional context

• Currently one Pod is mapped to one Branch ENI, if somehow mapping of more number of pods (of same deployment) to a same Branch ENI can also solve our use-case to improve the Pod Density.
• Only one Trunk ENI can be created per Host, if we allow multiple such Trunk ENI's (per host) then it can also help in further improving Pod density. This may not completely solve our problem since Pod Density would not increase drastically.

Example: C5.large / m5.large can have upto 9 Pods, if we double the Trunk ENI then still we can only launch 18 Pods.

Each pod (75m CPU request) X 18 Pods = 1350m CPU. So we still cannot fully utilize the host

[tsndqst]

Can we combine the ENABLE_PREFIX_DELEGATION feature from https://github.com/aws/containers-roadmap/issues/138 with Pod ENIs for higher pod density? It seems like it would be possible to enable prefix delegation on branch ENIs to get more IPs per branch ENI.

[mikestef9]

See answer here https://github.com/aws/containers-roadmap/issues/138#issuecomment-889242059

[macwork1]

Has there been any workarounds to the pod density limit when running with security groups enabled? SGs for PODs are quite elegant and the best way in my view to integrate Kubernetes with the AWS security model. Unfortunately the branch ENI limit prevents us from using this feature for almost all use cases. The only exception being one specific cluster where the POD memory footprints are large and we would reach the memory limits of the EC2 before the branch POD limit.

One of the points that I do not necessarily understand is why PODs that share the same security group cannot share the same branch ENI, but with secondary IP addresses per POD. We would typically want to apply a security group per namespace and the number of namespaces is low. Could secondary IP addresses be created for branch interfaces to reduce the number of ENIs created?