amazon-eks-ami Revisit kube-reserved calculation for containerd

Background

kube-reserved is meant to capture resource reservation for kubernetes system daemons (not including pods). For the amazon EKS ami it's mainly just the kubelet and container-runtime today. kube-reserved is typically a function of pod density on the nodes.

The current kube-reserved calculation is 2-3 years old (last updated as part of this experiment we did) when docker (+ shim) was the only container runtime option in the AMI. However, Amazon EKS ended support for Docker starting with the Kubernetes version 1.24 launch. Containerd is the only supported runtime now.

Here's the kube-reserved logic we have in our AMIs today: https://github.com/awslabs/amazon-eks-ami/blob/6b1df1107d6e7ec2f46612ebb509c16101ded22d/files/bootstrap.sh#L214 https://github.com/awslabs/amazon-eks-ami/blob/6b1df1107d6e7ec2f46612ebb509c16101ded22d/files/bootstrap.sh#L221-L225

Problem

Per the k8s containerd GA release blog the containerd integration consumes less CPU/memory in comparison with the docker integration (~10-12% - see below). We'll need to repeat the experiment given containerd versions being used in our AMIs are newer (1.5+). But overall we should expect notable savings.

At the steady state, with 105 pods, the containerd 1.1 integration consumes less CPU and memory overall compared to
Docker 18.03 CE integration with dockershim. The results vary with the number of pods running on the node, 105 is 
chosen because it is the current default for the maximum number of user pods per node.

As shown in the figures below, compared to Docker 18.03 CE integration with dockershim, the containerd 1.1
integration has 30.89% lower kubelet cpu usage, 68.13% lower container runtime cpu usage, 11.30% lower kubelet 
resident set size (RSS) memory usage, 12.78% lower container runtime RSS memory usage.

Additional Notes

With the recent launch of the Prefix Delegation (PD) feature in our AMIs, the supported pod densities have gone up for various instance types. The new logic sets max pod density as 110 for <= 30 vCPU instance-types and 250 for > 30 vCPU instance-types (xref). Discussed more in this issue. While PD should also be taken into account for kube-reserved calculation, that can happen independent from this change IMO since docker -> containerd switch is a common denominator. The drop in baseline usage will help customers reap incremental benefits, even if it means bumping it up later for PD-specific customers
Another issue is that - the bottlerocket AMI seems to have copied the same kube-reserved calculation as the EKS AMI (xrefs - cpu, mem) despite using containerd. Once we fix the calculation here, we should go have them update it too