pods with attached security groups cannot reach Pod Identity Agent link local address

[tmehlinger]

When using security groups for pods, pods with security groups attached cannot reach the Pod Identity agent on 169.254.170.23. Any pods without security groups can reach the agent without issue. The agent pods have no security groups associated and I've ensured that the security groups on failing pods permit egress traffic on TCP port 80, and my node security group permits ingress traffic on port 80 from cluster subnets. I've tried various combinations of egress/ingress from pod/node security groups, and even a blanket policy that permits traffic to/from 0/0 with no success.

I'm using the EKS Addon with the following configuration:

            {
                "enableNetworkPolicy": "true",
                "env": {
                    "ENABLE_POD_ENI": "true",
                },
                "init": {
                    "env": {
                        "DISABLE_TCP_EARLY_DEMUX": "true"
                    }
                }
            }

I've tried running the CNI with POD_SECURITY_GROUP_ENFORCING_MODE set to standard but this causes traffic to a peered VPCs to be denied in addition to pod identity traffic being dropped (and I want strict enforcement, regardless).

Reading the documentation for standard mode behavior:

inbound/outbound traffic from another pod on the same host or another service on the same host(such as kubelet/nodeLocalDNS) won't be enforced by security group rules.

My totally wild guess about what's happening is strict mode requires enforcement of security group rules and the node security group is dropping traffic destined for a link local address as invalid.

Could someone point me the right direction? Thanks!

Environment: Kubernetes version (use kubectl version): Server Version: version.Info{Major:"1", Minor:"28+", GitVersion:"v1.28.5-eks-5e0fdde", GitCommit:"e78a4be9da4c375a87a109e0f4a5f4a8d2bc17c0", GitTreeState:"clean", BuildDate:"2024-01-02T20:34:46Z", GoVersion:"go1.20.12", Compiler:"gc", Platform:"linux/amd64"}

CNI Version: v1.16.2-eksbuild.1

OS (e.g: cat /etc/os-release): AWS EKS 1.28.5 AMI.

AME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"
SUPPORT_END="2025-06-30"

Kernel (e.g. uname -a): Linux ip-10-0-128-192.us-west-2.compute.internal 5.10.205-195.807.amzn2.aarch64 #1 SMP Tue Jan 16 18:29:00 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux