AWS EKS connection issues

[sharedRoutine]

Bug Description

I believe this to be a bug, but I am not sure if this is just an incompatibility with AWS CNI for EKS given that the I got it working on other K8s (like microk8s).

The symptoms I get is that as soon as the redroid device has finished booting, networking stops working within the Pod. The only thing that is possible is pinging the Pod IP that the container lives in. KubeDNS Service ist at 172.20.0.10, the VPC used is 10.1.0.0/16, Pod IPs are within that range. The VPC has 3 public subnets in 3 availability zones. The public subnets have a public routing table and in general, even before redroid is fully booted, connections to the public internet and within the cluster work just fine.

Example Output Comparsion

Here is some output when emulator is running (busybox on a separate container within the same pod):

ping: bad address 'google.com'

/ # ping localhost
PING localhost (127.0.0.1): 56 data bytes
64 bytes from 127.0.0.1: seq=0 ttl=64 time=0.231 ms

ip r is empty

/ # route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
/ # 

/ # nslookup kube-dns
nslookup: can't connect to remote host (172.20.0.10): Network is unreachable

Same node, deployment etc. just without the emulator deployed:

/ # ping google.com
PING google.com (172.217.16.206): 56 data bytes

/ # route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         169.254.1.1     0.0.0.0         UG    0      0        0 eth0
169.254.1.1     0.0.0.0         255.255.255.255 UH    0      0        0 eth0
/ # 

/ # nslookup kube-dns
Server:		172.20.0.10
Address:	172.20.0.10:53

Reproducible Steps

I am using a custom AMI (ubuntu-eks/k8s_1.27/images/hvm-ssd/ubuntu-focal-20.04-arm64-server-20230809 ami-0427b3f3550be8018), therefore a custom launch template, which does not specify anything except for a user launch script that is necessary when using a custom AMI in EKS:

#!/bin/bash
/etc/eks/bootstrap.sh <YOUR CLUSTER NAME> \
 --use-max-pods false \
 --dns-cluster-ip 172.20.0.10 && \
 apt-get update -y && \
 apt-get install linux-modules-extra-`uname -r` -y && \
 modprobe binder_linux devices="binder,hwbinder,vndbinder" && \
 modprobe br_netfilter && \
 echo "options binder_linux devices=\"binder,hwbinder,vndbinder\"" >> /etc/modprobe.d/binder_linux.conf;

1. Create an EKS
2. Once the cluster has been initialized, add a node group with the launch template and add an instance
3. Once the instance has connected to the cluster, deploy a Pod with two containers: redroid/redroid:12.0.0_64only-latest and busybox

Redroid obviously needs

securityContext:
  privileged: true

Any help is appreciated, thanks