[BUG] "Exposing Services" guide does not work without IPv6

[dark-penguin]

If IPv6 support is disabled in the kernel, the guide does not work. After an investigation, I think I've found the problem:

$ kubectl get pod -n kube-system
NAME                                      READY   STATUS              RESTARTS   AGE
coredns-576bfc4dc7-pq2g6                  1/1     Running             0          129m
helm-install-traefik-bdtrg                0/1     Completed           2          129m
helm-install-traefik-crd-gz596            0/1     Completed           0          129m
local-path-provisioner-6795b5f9d8-2q9zl   1/1     Running             0          129m
metrics-server-557ff575fb-l46hq           1/1     Running             0          129m
svclb-traefik-3d8f13e3-6nxvt              0/2     ContainerCreating   0          128m
svclb-traefik-3d8f13e3-dstl7              0/2     ContainerCreating   0          128m
svclb-traefik-3d8f13e3-xndgm              0/2     ContainerCreating   0          128m
traefik-5fb479b77-2xqhs                   1/1     Running             0          128m

svclb-traefik-3d8f13e3 stuck in ContainerCreating?.. Interesting. Let's see...

$ kubectl events -n kube-system
3m27s (x26 over 8m50s)   Warning   FailedCreatePodSandBox   Pod/svclb-traefik-a96f858d-24pdl               Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: open /proc/sys/net/ipv6/conf/all/forwarding: no such file or directory: unknown
3m26s (x26 over 8m50s)   Warning   FailedCreatePodSandBox   Pod/svclb-traefik-a96f858d-hqvjl               Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: open /proc/sys/net/ipv6/conf/all/forwarding: no such file or directory: unknown
3m24s (x26 over 8m50s)   Warning   FailedCreatePodSandBox   Pod/svclb-traefik-a96f858d-2b8qs               Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: open /proc/sys/net/ipv6/conf/all/forwarding: no such file or directory: unknown

So this is because I follow the best security practices and disable IPv6 support completely if I don't need it, especially in IoT, which is kinda one of the main targets for k3s. :slightly_smiling_face:

Is there maybe at least a way to configure the built-in svclb to not even try IPv6 ? :slightly_smiling_face:

What did you do

Exactly followed the first part of this guide: https://k3d.io/v5.7.3/usage/exposing_services/#1-via-ingress-recommended

• How was the cluster created?

  • k3d cluster create --api-port 6550 -p "8081:80@loadbalancer" --agents 2

• What did you do afterwards?

  • kubectl create deployment nginx --image=nginx
  • kubectl create service clusterip nginx --tcp=80:80
  • nano thatfile.yaml # Paste the YAML in the guide
  • kubectl apply -f thatfile.yaml
  • curl localhost:8081

What did you expect to happen

curl outputting the default Nginx HTML page

Actual output:

curl: (52) Empty reply from server

Which OS & Architecture

$ k3d runtime-info
arch: x86_64
cgroupdriver: systemd
cgroupversion: "2"
endpoint: /var/run/docker.sock
filesystem: extfs
infoname: ASUS
name: docker
os: Debian GNU/Linux 12 (bookworm)
ostype: linux
version: 26.1.3

Which version of k3d

$ k3d version
k3d version v5.7.3
k3s version v1.30.3-k3s1 (default)

Which version of docker

$ docker verson
docker: 'verson' is not a docker command.
See 'docker --help'

$ docker info
Client: Docker Engine - Community
 Version:    26.1.3
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.14.0
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.27.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 5
  Running: 5
  Paused: 0
  Stopped: 0
 Images: 5
 Server Version: 26.1.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: false
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8b3b7ca2e5ce38e8f31a34f35b2b68ceb8470d89
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.9.3-1-liquorix-amd64
 Operating System: Debian GNU/Linux 12 (bookworm)
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.33GiB
 Name: ASUS
 ID: b2908df4-2b26-4e7f-b9a6-2e7ecd840319
 Docker Root Dir: /hdd/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

[dark-penguin]

To fix at runtime:

kubectl patch -n kube-system daemonsets.apps "$(kubectl get -n kube-system daemonsets.apps | awk '/svclb/ {print $1;}')" -p '{"spec": {"template": {"spec": {"securityContext": {"sysctls": [{"name": "net.ipv4.ip_forward", "value": "1"}]}}}}}'

This is the easiest way I've found to remove the second item in that list, which is the problem:

- name: net.ipv6.conf.all.forwarding
  value: "1"

Would it be possible to configure that to be done automatically?

[camme]

I have the same issue with curl: (52) Empty reply from server but dont have the row

- name: net.ipv6.conf.all.forwarding
  value: "1"

Followed the guide but cant seem to fix curl: (52) Empty reply from server.

[yakom]

@dark-penguin hi, i've tried to reproduce the issue against k3d v5.8.1 (and a different OS but with IPv6 disabled and /proc/sys/net/ipv6/ missing). everything works fine. if the issue is still relevant, could you check against the newest version?

[dark-penguin]

Ho ho ho! 🎅 Works fine in 5.8.1 ! 🥳 Thank you!