Stijn De Haes

> Thanks for testing this out, my findings concur. I'll see what I can do! @cartermckinnon thank you very much, if I can help with anything please reach out :)...

Just retried it, the result are still similar. The biggest issue appears to be the call to get the version of kubelet, if we could get the version another way...

@dims correct! That is the reason for the big timeout between these 2 log lines: ``` May 22 14:03:48.312560 localhost nodeadm[1689]: {"level":"info","ts":1716386628.3125567,"caller":"init/init.go:103","msg":"Configuring daemon...","name":"kubelet"} May 22 14:04:01.246690 localhost nodeadm[1689]: {"level":"info","ts":1716386641.2466207,"caller":"kubelet/config.go:300","msg":"Detected kubelet...

I did the following experiment. I change the `GetKubeletVersion` function to the following: ``` func GetKubeletVersion() (string, error) { if kubeletVersion := os.Getenv("KUBELET_VERSION"); kubeletVersion != "" { zap.L().Info("Found kubelet version...

As an FYI here is the systemd-analyze plot from an AL2 Image. Startup finished in 1.355s (kernel) + 1.379s (initrd) + 18.876s (userspace) = 21.611s As you can see the...

> @stijndehaes looking at the graphs for al2023 above, what happens if you disable the update-motd.service? (the before picture shows significant contribution from that one) Just disable update-motd.service results in:...

> @stijndehaes another one, what happens if you switch the order of getting kubelet version first (caching it) and THEN configuring the kubelet service? I am not exactly sure if...

@dims made a PR for disabling motd. https://github.com/awslabs/amazon-eks-ami/pull/1910 As to your suggestion to cache the kubelet version the biggest issue is that we are still using the kubelet executable and...

We are also running into this issue. It introduces flakes in our own controller loop luckily we have automatic retries. @abhipth if you need someone to test the solution I...