common
common copied to clipboard
containers
Usernetes doesn't work with Pasta mode of Podman (works fine with slirp4netns mode): `Get "https://10.96.0.1:443/api/v1/namespaces/kube-flannel/pods/kube-flannel-ds-pnnrt": dial tcp 10.96.0.1:443: i/o timeout`
Usernetes (Kubernetes in Rootless Docker/Podman/nerdctl) works fine with Rootless Podman v5 + slirp4netns.
However, it doesn't seem to work with Pasta:
$ kubectl get -n kube-flannel pods
NAME READY STATUS RESTARTS AGE
kube-flannel-ds-pnnrt 0/1 CrashLoopBackOff 7 (58s ago) 15m
: ↑ kubectl can connect to kube-apiserver
$ kubectl logs -n kube-flannel daemonsets/kube-flannel-ds
Defaulted container "kube-flannel" out of: kube-flannel, install-cni-plugin (init), install-cni (init)
Error from server: Get "https://192.168.5.15:10250/containerLogs/kube-flannel/kube-flannel-ds-pnnrt/kube-flannel": dial tcp 192.168.5.15:10250: i/o timeout
: ↑ kube-apiserver is failing to connect to kubelet
$ podman exec usernetes_node_1 sh -euxc 'cat /var/log/containers/kube-flannel-ds-*_kube-flannel_kube-flannel-*.log'
+ cat /var/log/containers/kube-flannel-ds-pnnrt_kube-flannel_kube-flannel-81d4059f4344ffb796b1ac0de247cf71d4b5dfc837a03e0307a54103e8e618ed.log
2024-12-02T20:59:22.893780871Z stderr F I1202 20:59:22.892812 1 main.go:212] CLI flags config: {etcdEndpoints:http://127.0.0.1:4001,http://127.0.0.1:2379 etcdPrefix:/coreos.com/network etcdKeyfile: etcdCertfile: etcdCAFile: etcdUsername: etcdPassword: version:false kubeSubnetMgr:true kubeApiUrl: kubeAnnotationPrefix:flannel.alpha.coreos.com kubeConfigFile: iface:[] ifaceRegex:[] ipMasq:true ifaceCanReach: subnetFile:/run/flannel/subnet.env publicIP: publicIPv6: subnetLeaseRenewMargin:60 healthzIP:0.0.0.0 healthzPort:0 iptablesResyncSeconds:5 iptablesForwardRules:true netConfPath:/etc/kube-flannel/net-conf.json setNodeNetworkUnavailable:true}
2024-12-02T20:59:22.893836867Z stderr F W1202 20:59:22.893326 1 client_config.go:618] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
2024-12-02T20:59:52.908415802Z stderr F E1202 20:59:52.908036 1 main.go:229] Failed to create SubnetManager: error retrieving pod spec for 'kube-flannel/kube-flannel-ds-pnnrt': Get "https://10.96.0.1:443/api/v1/namespaces/kube-flannel/pods/kube-flannel-ds-pnnrt": dial tcp 10.96.0.1:443: i/o timeout
: ↑ the flannel pod is failing to connect to KUBERNETES_SERVICE_HOST
I haven't figured out whether this is Podman's misconfiguration of pasta, or a bug of pasta itself. I'm opening an issue here anyway so as to inform that Podman shouldn't drop the support for slirp4netns yet.
Reproduction steps
- Set
network.default_rootless_network_cmdto "pasta" or "slirp4netns"
mkdir -p "$HOME/.config/containers/containers.conf.d"
cat <<EOF >"$HOME/.config/containers/containers.conf.d/network.conf"
[network]
# "pasta" (default since Podman v5) or "slirp4netns"
default_rootless_network_cmd="slirp4netns"
EOF
- Install Podman, Podman Compose, and misc utilities
sudo dnf install -y podman podman-compose git make jq kubectl
- Configure cgroup v2 delegation
sudo mkdir -p /etc/systemd/system/[email protected]
sudo tee /etc/systemd/system/[email protected]/delegate.conf <<EOF >/dev/null
[Service]
Delegate=cpu cpuset io memory pids
EOF
sudo systemctl daemon-reload
- Load kernel modules
sudo modprobe br_netfilter
sudo modprobe vxlan
- Set up a node of Usernetes using Rootless Podman
git clone https://github.com/rootless-containers/usernetes.git
cd usernetes
git checkout gen2-v20241203.0
export CONTAINER_ENGINE=podman
make up
make kubeadm-init
make install-flannel
make kubeconfig
export KUBECONFIG="$(pwd)/kubeconfig"
- Check the status of
kube-flannel-ds
🔴 pasta (CrashLoopBackOff):
$ kubectl get -n kube-flannel pods
NAME READY STATUS RESTARTS AGE
kube-flannel-ds-pnnrt 0/1 CrashLoopBackOff 7 (58s ago) 15m
: ↑ kubectl can connect to kube-apiserver
$ kubectl logs -n kube-flannel daemonsets/kube-flannel-ds
Defaulted container "kube-flannel" out of: kube-flannel, install-cni-plugin (init), install-cni (init)
Error from server: Get "https://192.168.5.15:10250/containerLogs/kube-flannel/kube-flannel-ds-pnnrt/kube-flannel": dial tcp 192.168.5.15:10250: i/o timeout
: ↑ kube-apiserver is failing to connect to kubelet
$ podman exec usernetes_node_1 sh -euxc 'cat /var/log/containers/kube-flannel-ds-*_kube-flannel_kube-flannel-*.log'
+ cat /var/log/containers/kube-flannel-ds-pnnrt_kube-flannel_kube-flannel-81d4059f4344ffb796b1ac0de247cf71d4b5dfc837a03e0307a54103e8e618ed.log
2024-12-02T20:59:22.893780871Z stderr F I1202 20:59:22.892812 1 main.go:212] CLI flags config: {etcdEndpoints:http://127.0.0.1:4001,http://127.0.0.1:2379 etcdPrefix:/coreos.com/network etcdKeyfile: etcdCertfile: etcdCAFile: etcdUsername: etcdPassword: version:false kubeSubnetMgr:true kubeApiUrl: kubeAnnotationPrefix:flannel.alpha.coreos.com kubeConfigFile: iface:[] ifaceRegex:[] ipMasq:true ifaceCanReach: subnetFile:/run/flannel/subnet.env publicIP: publicIPv6: subnetLeaseRenewMargin:60 healthzIP:0.0.0.0 healthzPort:0 iptablesResyncSeconds:5 iptablesForwardRules:true netConfPath:/etc/kube-flannel/net-conf.json setNodeNetworkUnavailable:true}
2024-12-02T20:59:22.893836867Z stderr F W1202 20:59:22.893326 1 client_config.go:618] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
2024-12-02T20:59:52.908415802Z stderr F E1202 20:59:52.908036 1 main.go:229] Failed to create SubnetManager: error retrieving pod spec for 'kube-flannel/kube-flannel-ds-pnnrt': Get "https://10.96.0.1:443/api/v1/namespaces/kube-flannel/pods/kube-flannel-ds-pnnrt": dial tcp 10.96.0.1:443: i/o timeout
: ↑ the flannel pod is failing to connect to KUBERNETES_SERVICE_HOST
🟢 slirp4netns (Running):
$ kubectl get -n kube-flannel pods
NAME READY STATUS RESTARTS AGE
kube-flannel-ds-cbkh4 1/1 Running 0 13m
$ kubectl logs -n kube-flannel daemonsets/kube-flannel-ds
Defaulted container "kube-flannel" out of: kube-flannel, install-cni-plugin (init), install-cni (init)
I1202 20:54:04.703825 1 main.go:212] CLI flags config: {etcdEndpoints:http://127.0.0.1:4001,http://127.0.0.1:2379 etcdPrefix:/coreos.com/network etcdKeyfile: et
cdCertfile: etcdCAFile: etcdUsername: etcdPassword: version:false kubeSubnetMgr:true kubeApiUrl: kubeAnnotationPrefix:flannel.alpha.coreos.com kubeConfigFile: iface:[
] ifaceRegex:[] ipMasq:true ifaceCanReach: subnetFile:/run/flannel/subnet.env publicIP: publicIPv6: subnetLeaseRenewMargin:60 healthzIP:0.0.0.0 healthzPort:0 iptables
ResyncSeconds:5 iptablesForwardRules:true netConfPath:/etc/kube-flannel/net-conf.json setNodeNetworkUnavailable:true}
W1202 20:54:04.704890 1 client_config.go:618] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I1202 20:54:04.733564 1 kube.go:139] Waiting 10m0s for node controller to sync
I1202 20:54:04.733734 1 kube.go:469] Starting kube subnet manager
I1202 20:54:05.734597 1 kube.go:146] Node controller sync successful
I1202 20:54:05.734862 1 main.go:232] Created subnet manager: Kubernetes Subnet Manager - u7s-lima-vm1
I1202 20:54:05.734880 1 main.go:235] Installing signal handlers
I1202 20:54:05.735058 1 main.go:469] Found network config - Backend type: vxlan
[...]
I1202 20:54:05.889340 1 main.go:413] Wrote subnet file to /run/flannel/subnet.env
I1202 20:54:05.889892 1 main.go:417] Running backend.
I1202 20:54:05.890676 1 vxlan_network.go:65] watching for new subnet leases
I1202 20:54:05.913585 1 main.go:438] Waiting for all goroutines to exit
I1202 20:54:05.923924 1 iptables.go:372] bootstrap done
I1202 20:54:05.938226 1 iptables.go:372] bootstrap done
Host Environment
podman-5.2.5-1.fc41.x86_64
podman-compose-1.2.0-2.fc41.noarch
passt-0^20240906.g6b38f07-1.fc41.x86_64
passt-selinux-0^20240906.g6b38f07-1.fc41.noarch
libslirp-4.8.0-2.fc41.x86_64
slirp4netns-1.3.1-1.fc41.x86_64
