CNI image not reflecting cluster region

[ztmr]

I am working with a client who is blocking most of the regions other than eu-west-2.

Spinning up a new EKS cluster in eu-west-2 ends up with all nodes in NotReady state. Checking status of the aws-node-* pods indicates that the CNI image location is us-west-2, which is blocked, and the pod is therefore failing on ImagePullBackOff and nodes are never transitioning to the Ready state.

Manually changing the aws-node DaemonSet to use eu-west-2 ECR fixes everything immediately.

Since the cluster is not ready, Pulumi does not carry on installing any K8s resources -- which would be a workaround for patching this quickly. What's even worse, is that it does not even expose the kubeconfig output to allow for any manual intervention.

I have looked through the code (https://github.com/pulumi/pulumi-eks/blob/v0.30.0/nodejs/eks/cni/aws-k8s-cni.yaml and https://github.com/pulumi/pulumi-eks/blob/v0.30.0/nodejs/eks/cmd/provider/cni.ts) and there is an option to pass a custom image which only affects the main pod container, but not the initContainers...

And even doing this is not a nice solution because we just want to bypass the region, not fiddle with the version numbers -- we want to use the version that you recommend without having to maintain it separately.

I would therefore suggest two things:

• introduce a new option ecrRegion
  • if this is not specified, fall back to the region of the cluster itself
• either dump the image option entirely, or at least make it more flexible:
  • have complex field of initImage and image allowing to pass name and tag fields

Is there any workaround for this in the meantime?

[lukehoban]

The CNI YAML we use here is based on what's available from upstream. The upstream does indeed hard code this in, and expects that to be accessible. We could definitely add additional configuration options to make this easier to override. EKS itself also installs and manages this themselves, so you may be able to avoid this by skipping managing CNI via the Pulumi EKS component (useDefaultVpcCni). The managed CNI add-on being tracked in https://github.com/pulumi/pulumi-eks/issues/587 might also help here when that is added.