eksctl icon indicating copy to clipboard operation
eksctl copied to clipboard

[Bug] eksctl update-aws-node updated aws-eks-nodeagent to an invalid image in AWS GovCloud

Open 1riggs opened this issue 1 year ago • 2 comments

What were you trying to accomplish?

As part of administering an EKS Cluster in AWS GovCloud, us-gov-west-1, I upgraded the cluster version (from v1.26 -> v1.27). As per the documentation I was updating the addons, including aws-node. To that end, I ran eksctl utils update-aws-node --cluster my-cluster.

I expected the aws-node DaemonSet to update with the appropriate container versions, as documented in the eksctl documentation.

What happened?

Updated configuration included 602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon/aws-network-policy-agent:v1.1.1. This results in errors when pulling that image for the aws-eks-nodeagent:

ImagePullBackOff (Back-off pulling image "602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon/aws-network-policy-agent:v1.1.1")

How to reproduce it?

Run the aforementioned command after a cluster version update in the AWS GovCloud region, us-gov-west-1.

Logs

N/A

Anything else we need to know?

Suspect the registry/region for the aws-eks-nodeagent is incorrect. The other apps in that DaemonSet (aws-node & aws-vpc-cni-init) use 013241004608.dkr.ecr.us-gov-west-1.amazonaws.com.

Versions

$ eksctl info
eksctl version: 0.192.0
kubectl version: v1.25.16
OS: linux

1riggs avatar Oct 15 '24 03:10 1riggs

Hello 1riggs :wave: Thank you for opening an issue in eksctl project. The team will review the issue and aim to respond within 1-5 business days. Meanwhile, please read about the Contribution and Code of Conduct guidelines here. You can find out more information about eksctl on our website

github-actions[bot] avatar Oct 15 '24 03:10 github-actions[bot]

Manually modifying the DeamonSet to use 013241004608.dkr.ecr.us-gov-west-1.amazonaws.com/amazon/aws-network-policy-agent:v1.1.1 does work.

1riggs avatar Oct 15 '24 03:10 1riggs

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

github-actions[bot] avatar Nov 15 '24 02:11 github-actions[bot]

This issue was closed because it has been stalled for 5 days with no activity.

github-actions[bot] avatar Nov 21 '24 02:11 github-actions[bot]

I ran into this same issue running:

eksctl utils update-aws-node --cluster=${cluster_name} --approve

The image gets updated to image: 602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon/aws-network-policy-agent:v1.1.1 But the cluster is in eu-west-1 so the image pull fails until you update the daemonset manually.

λ eksctl info
eksctl version: 0.194.0
kubectl version: v1.31.2
OS: windows

rachekalmir avatar Dec 01 '24 19:12 rachekalmir

Confirmed same problem on:

eksctl info
eksctl version: 0.197.0
kubectl version: v1.30.6-dispatcher
OS: linux

rachekalmir avatar Dec 04 '24 19:12 rachekalmir

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

github-actions[bot] avatar Jan 13 '25 02:01 github-actions[bot]

This issue was closed because it has been stalled for 5 days with no activity.

github-actions[bot] avatar Jan 19 '25 02:01 github-actions[bot]