Inner Docker Network Connectivity Issues

[christensenjairus]

We are running Ubuntu 24.04 LTS hosts with 1.32 K8s. We've followed the installation manifests and have sysbox-deploy-k8s running on each worker node with no issues/errors. We have made no edits to the mgr or fs settings of the sysbox configmap. K8s uses cilium in full ebpf mode, replacing kube-proxy, if it's relevant.

Inside an ubuntu container using the nestybox docker or docker/systemd images, we cannot seem to get docker networking to work correctly between inner containers.

We've made sure to set the runtimeClass, io.kubernetes.cri-o.userns-mode annotation, and even the container.seccomp.security.alpha.kubernetes.io/dev and container.apparmor.security.beta.kubernetes.io/dev annotations.

Here's a simple example using docker-compose

version: '3.9'

services:
  alpine:
    image: alpine:latest
    command: tail -f /dev/null
    networks:
      - mynet

  nginx:
    image: nginx:latest
    ports:
      - "80:80"
    networks:
      - mynet

networks:
  mynet:

Container-to-container connectivity will work for ICMP (tested with ping) and TCP (tested with netcat), but not using other protocols. If we try to wget or curl the nginx container from the alpine container, we'll get a an error saying server returned error: HTTP/1.1 503 Service Unavailable. This is not, however, an nginx issue, as this error occurs on many different services and types of services. We noticed this because we couldn't get a graphql container to talk to a postgres container while running nhost up.

Networking to the outer container, however, remains unbroken. We can curl nginx fine using localhost:80 from the outer container.

Please let us know what else we can try. I've been trying to get this to work for 2-3 days and I'm losing my mind 😂

[ctalledo]

Hi @christensenjairus, apologies for the belated reply and thank you for giving Sysbox a shot in your K8s cluster.

K8s uses cilium in full ebpf mode, replacing kube-proxy, if it's relevant.

I have a hunch this is the culprit, though theoretically it shouldn't be an issue because Cilium handles pod-to-pod networking and should not interfere with intra-pod networking as you are trying to do here. IIUC, you have a single pod deployed with Sysbox, and inside that pod you run docker engine plus which starts a couple of (nested) containers, but they can't communicate with each other except with ICMP and TCP.

If possible, I would suggest trying without Cilium (just use kube-proxy) and see if the problem reproduces.

FYI, Sysbox stays pretty much out of the way when it comes to container networking, letting Docker or Kubernetes set it up.

Thanks!