ipamd | Failed to delete eniConfig

[sebastianserrano]

trafficstars

What happened: ipamd failed to delete eniConfig while bootstrapping cluster

Attach logs Logs from /var/log/aws-routed-eni/ipamd.log

{"level":"error","ts":"2024-06-06T12:16:45.771Z","caller":"ipamd/ipamd.go:415","msg":"Failed to delete eniConfig node label%!(EXTRA *errors.StatusError=nodes \"ip-10-0-12-183.eu-north-1.compute.internal\" is forbidden: User \"system:serviceaccount:kube-system:aws-node\" cannot update resource \"nodes\" in API group \"\" at the cluster scope)"}
{"level":"error","ts":"2024-06-06T12:16:45.771Z","caller":"aws-k8s-agent/main.go:27","msg":"Initialization failure: nodes \"ip-10-0-12-183.eu-north-1.compute.internal\" is forbidden: User \"system:serviceaccount:kube-system:aws-node\" cannot update resource \"nodes\" in API group \"\" at the cluster scope"}

Logs from bash /opt/cni/bin/aws-cni-support.sh

What you expected to happen: Expected the CNI to bootstrap the network layer in the cluster

How to reproduce it (as minimally and precisely as possible): Unfortunately, this happens intermittently

Environment:

• Kubernetes version (use kubectl version):
  • 1.28
• CNI Version
  • v1.18.1-eksbuild.3
• OS (e.g: cat /etc/os-release):

NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"
SUPPORT_END="2025-06-30"

• Kernel (e.g. uname -a):
  • Linux 5.10.217-205.860.amzn2.x86_64 #1 SMP Tue May 21 16:52:24 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Notes: Of course, one could argue that the simply the solution is to add the ability to delete ENIs to the aws-node cluster role but is that the right way?