Measure Pod Start & Node Join Performance for k8s Variants

[cbgbt]

We are moving to containerd 1.6.4 for Bottlerocket in https://github.com/bottlerocket-os/bottlerocket/pull/2158. We have also made improvements to the kubelet config in https://github.com/bottlerocket-os/bottlerocket/pull/2166 which should improve Pod start time.

This issue is to track testing both of these changes to assert that performance has improved or remained the same for Node Join and Pod Start with both of these changes.

[mikkoc]

Is there an easy way to test 1.8.0, like a nightly AMI or similar?

The pods startup time for bottlerocket is currently abysmal compared to our regular AmazonLinux2 AMIs:

[cbgbt]

Hi @mikkoc. Sorry for the delayed response. We don't currently have a nightly AMI to test, so the easiest way to do the test would be with a Bottlerocket AMI built from the develop branch.

I'm in the process of gathering my own metrics here, but some preliminary testing of the changes made in #2166 are quite promising.

[cbgbt]

I ran out of time to complete this prior to the Bottlerocket 1.8.0 release. We've been discussing ways to use the Prometheus node-exporter to get more data than my simple automation was gathering, so I'll keep this issue open to track extracting metrics in that way.

[cbgbt]

Closing this in favor of #2747 for now.