ovn-kubernetes
ovn-kubernetes copied to clipboard
ovn-org
L3-UDN: Failure to create VM's network
What happened?
For this case, we have applied the following Layer 3 topology
apiVersion: v1 kind: Namespace metadata: name: udn-test labels: k8s.ovn.org/primary-user-defined-network: "" --- apiVersion: k8s.ovn.org/v1 kind: UserDefinedNetwork metadata: name: l3-primary namespace: udn-test spec: topology: Layer3 layer3: role: Primary subnets: - cidr: 10.20.0.0/16
We have created a container and a VM with an L2Bridge as the interface. The UDN has correctly created both the vRouter and the vSwitch, installing the defaults properly.
2.1 Container
bash-5.0# ip add 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0@if33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default link/ether 0a:58:0a:2b:00:05 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10.43.0.5/24 brd 10.43.0.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::858:aff:fe2b:5/64 scope link valid_lft forever preferred_lft forever 3: ovn-udn1@if34: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default link/ether 0a:58:0a:14:00:05 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10.20.0.5/24 brd 10.20.0.255 scope global ovn-udn1 valid_lft forever preferred_lft forever inet6 fe80::858:aff:fe14:5/64 scope link valid_lft forever preferred_lft forever bash-5.0# ip route default via 10.20.0.1 dev ovn-udn1 10.20.0.0/24 dev ovn-udn1 proto kernel scope link src 10.20.0.5 10.20.0.0/16 via 10.20.0.1 dev ovn-udn1 10.43.0.0/24 dev eth0 proto kernel scope link src 10.43.0.5 10.43.0.0/16 via 10.43.0.1 dev eth0 10.96.0.0/16 via 10.20.0.1 dev ovn-udn1 100.64.0.0/16 via 10.43.0.1 dev eth0 100.65.0.0/16 via 10.20.0.1 dev ovn-udn1 bash-5.0# ping 8.8.8.8 PING 8.8.8.8 (8.8.8.8): 56 data bytes 64 bytes from 8.8.8.8: seq=0 ttl=110 time=50.000 ms 64 bytes from 8.8.8.8: seq=1 ttl=110 time=53.685 ms
It works correctly; the only thing I'd like to understand is whether it's possible to mask the IP of eth0 from the user performing SSH towards the default cluster.
2.2 VM from virctl console
root@vm-test:/home/ubuntu# ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc fq_codel state UP group default qlen 1000 link/ether 0a:58:0a:14:01:05 brd ff:ff:ff:ff:ff:ff inet6 fe80::858:aff:fe14:105/64 scope link valid_lft forever preferred_lft forever root@vm-test:/home/ubuntu# ip route root@vm-test:/home/ubuntu# ip nei
The NIC does not receive the IP, and the NIC appears different from the one we see in the describe POD
annotations: │
│ descheduler.alpha.kubernetes.io/request-evict-only: "" │
│ k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.43.2.13/24"],"mac_address":"0a:58:0a:2b:02:0d","routes":[{"dest":"10.43.0.0/16","nextHop":"10.43.2.1"},{"dest":"100.64.0.0/16","nextHop":"10.43.2.1"}],"ip_a │
│ ddress":"10.43.2.13/24","role":"infrastructure-locked"},"udn-test/l3-primary":{"ip_addresses":["10.20.1.5/24"],"mac_address":"0a:58:0a:14:01:05","gateway_ips":["10.20.1.1"],"routes":[{"dest":"10.20.0.0/16","nextHop":"10 │
│ .20.1.1"},{"dest":"10.96.0.0/16","nextHop":"10.20.1.1"},{"dest":"100.65.0.0/16","nextHop":"10.20.1.1"}],"ip_address":"10.20.1.5/24","gateway_ip":"10.20.1.1","role":"primary"}}' │
│ k8s.v1.cni.cncf.io/network-status: |- │
│ [{ │
│ "name": "ovn-kubernetes", │
│ "interface": "eth0", │
│ "ips": [ │
│ "10.43.2.13" │
│ ], │
│ "mac": "0a:58:0a:2b:02:0d", │
│ "dns": {} │
│ },{ │
│ "name": "ovn-kubernetes", │
│ "interface": "ovn-udn1", │
│ "ips": [ │
│ "10.20.1.5" │
│ ], │
│ "mac": "0a:58:0a:14:01:05", │
│ "default": true, │
│ "dns": {} │
│ }]
It seems that the routing table is not visible from the console but can be viewed from the shell in k9s.
2.3 VM from k9s shell as qemu user login:
bash-5.1$ ip add 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host proto kernel_lo valid_lft forever preferred_lft forever 2: eth0@if36: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default link/ether 0a:58:0a:2b:02:0d brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10.43.2.13/24 brd 10.43.2.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::858:aff:fe2b:20d/64 scope link proto kernel_ll valid_lft forever preferred_lft forever 3: ovn-udn1-nic@if37: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master k6t-ovn-udn1 state UP group default link/ether 92:a3:39:0d:e2:e7 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet6 fe80::90a3:39ff:fe0d:e2e7/64 scope link proto kernel_ll valid_lft forever preferred_lft forever 4: k6t-ovn-udn1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default qlen 1000 link/ether 42:6f:ea:d2:6e:6f brd ff:ff:ff:ff:ff:ff inet6 fe80::90a3:39ff:fe0d:e2e7/64 scope link proto kernel_ll valid_lft forever preferred_lft forever 5: tap0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc fq_codel master k6t-ovn-udn1 state UP group default qlen 1000 link/ether 42:6f:ea:d2:6e:6f brd ff:ff:ff:ff:ff:ff inet6 fe80::406f:eaff:fed2:6e6f/64 scope link proto kernel_ll valid_lft forever preferred_lft forever 6: ovn-udn1: <BROADCAST,NOARP> mtu 1400 qdisc noop state DOWN group default qlen 1000 link/ether 0a:58:0a:14:01:05 brd ff:ff:ff:ff:ff:ff inet 10.20.1.5/24 brd 10.20.1.255 scope global ovn-udn1 valid_lft forever preferred_lft forever inet6 fe80::858:aff:fe14:105/64 scope link valid_lft forever preferred_lft forever bash-5.1$ ip route 10.43.0.0/16 via 10.43.2.1 dev eth0 10.43.2.0/24 dev eth0 proto kernel scope link src 10.43.2.13 100.64.0.0/16 via 10.43.2.1 dev eth0
Unlike the Layer2 topology ISSUE, routes are not installed and the port ovn-udn1 is DOWN.
2.4 LOG UDN creation
What did you expect to happen?
We expected the automatic creation of the gateway router in the layer 3 topology only for virtual machines.
How can we reproduce it (as minimally and precisely as possible)?
Installation of a K3s cluster with the following parameters:
server \
'--cluster-init' \
'--disable-cloud-controller' \
'--disable-network-policy' \
'--flannel-backend=none' \
'--disable' \
'servicelb' \
'--disable' \
'traefik' \
'--disable-kube-proxy' \
Installation of ovn-kubernetes using Helm:
helm install ovn-kubernetes . -f values-no-ic.yaml \
--set k8sAPIServer="https://10.60.0.7:6443" \
--set ovnkube-identity.replicas=1 \
--set global.image.repository=ghcr.io/ovn-kubernetes/ovn-kubernetes/ovn-kube-ubuntu \
--set global.image.tag=master \
--set global.enableOvnKubeIdentity=false \
--set podNetwork=10.43.0.0/16/24 \
--set serviceNetwork=10.96.0.0/16 \
--set global.enableMultiNetwork=true \
--set global.enablePersistentIPs=true \
--set global.enableNetworkSegmentation=true
Creation of the L3 UDN:
apiVersion: v1
kind: Namespace
metadata:
name: udn-test
labels:
k8s.ovn.org/primary-user-defined-network: ""
---
apiVersion: k8s.ovn.org/v1
kind: UserDefinedNetwork
metadata:
name: l3-primary
namespace: udn-test
spec:
topology: Layer3
layer3:
role: Primary
subnets:
- cidr: 10.20.0.0/16
Anything else we need to know?
No response
OVN-Kubernetes version
$ ovnkube --version
1.0.0
Kubernetes version
$ kubectl version
Client Version: v1.31.6+k3s1
Kustomize Version: v5.4.2
Server Version: v1.31.6+k3s1
OVN version
$ oc rsh -n ovn-kubernetes ovnkube-node-xxxxx (pick any ovnkube-node pod on your cluster)
$ rpm -q ovn
# paste output here
OVS version
$ oc rsh -n ovn-kubernetes ovs-node-xxxxx (pick any ovs pod on your cluster)
$ rpm -q openvswitch
# paste output here
Platform
Ubuntu Server running on VMs managed by proxmox
OS version
# On Linux:
$ cat /etc/os-release
NAME="Ubuntu"
VERSION_ID="24.10"
VERSION="24.10 (Oracular Oriole)"
$ uname -a
Linux k3s-master 6.11.0-19-generic #19-Ubuntu SMP PREEMPT_DYNAMIC Wed Feb 12 21:43:43 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Install tools
Container runtime (CRI) and version (if applicable)
Thanks for the report.. We'll try to take a look at this in our next community meeting.
@PGhiorzo
May I know which helm repo you add for ovn-kubernetes because I got something broken as follows
helm install ovn-kubernetes . -f values-no-ic.yaml \
--set k8sAPIServer="https://192.168.0.124:6443" \
--set ovnkube-identity.replicas=1 \
--set global.image.repository=ghcr.io/ovn-kubernetes/ovn-kubernetes/ovn-kube-ubuntu \
--set global.image.tag=master \
--set podNetwork=10.96.0.0/16/24 \
--set serviceNetwork=172.17.17.0/24 \
--set global.enableOvnKubeIdentity=false \
--set global.enablePersistentIPs=true \
--set global.enableMultiNetwork=true \
--set global.enableNetworkSegmentation=true
walk.go:75: found symbolic link in path: /root/ovn-kubernetes/helm/ovn-kubernetes/crds/k8s.ovn.org_adminpolicybasedexternalroutes.yaml resolves to /root/ovn-kubernetes/dist/templates/k8s.ovn.org_adminpolicybasedexternalroutes.yaml.j2. Contents of linked file included and used
walk.go:75: found symbolic link in path: /root/ovn-kubernetes/helm/ovn-kubernetes/crds/k8s.ovn.org_egressfirewalls.yaml resolves to /root/ovn-kubernetes/dist/templates/k8s.ovn.org_egressfirewalls.yaml.j2. Contents of linked file included and used
walk.go:75: found symbolic link in path: /root/ovn-kubernetes/helm/ovn-kubernetes/crds/k8s.ovn.org_egressips.yaml resolves to /root/ovn-kubernetes/dist/templates/k8s.ovn.org_egressips.yaml.j2. Contents of linked file included and used
walk.go:75: found symbolic link in path: /root/ovn-kubernetes/helm/ovn-kubernetes/crds/k8s.ovn.org_egressqoses.yaml resolves to /root/ovn-kubernetes/dist/templates/k8s.ovn.org_egressqoses.yaml.j2. Contents of linked file included and used
walk.go:75: found symbolic link in path: /root/ovn-kubernetes/helm/ovn-kubernetes/crds/k8s.ovn.org_egressservices.yaml resolves to /root/ovn-kubernetes/dist/templates/k8s.ovn.org_egressservices.yaml.j2. Contents of linked file included and used
NAME: ovn-kubernetes
LAST DEPLOYED: Sat Mar 29 04:39:38 2025
NAMESPACE: ovn-kubernetes
STATUS: deployed
REVISION: 1
TEST SUITE: None
Hi @cloudcafetech , we are applied the above configurations starting from a branch git repository fixed by @flavio-fernandes : https://github.com/flavio-fernandes/ovn-kubernetes/tree/helm-with-network-segmentation As I can see, on your env config there isn't the parameter OVN_NETWORK_SEGMENTATION_ENABLE = "true" Maybe you can re-deploy the ovn cni with this branch and you report other issues. Keep in mind that me and @PGhiorzo are posting two bug report after we solved your similar issues:
- [L3-UDN: Failure to create VM's network](https://github.com/ovn-kubernetes/ovn-kubernetes/issues/5155)
- [L2-UDN: Failure to create network](https://github.com/ovn-kubernetes/ovn-kubernetes/issues/5154)
@garca-cloud Thanks for updates.
I did with flavio' helm but not able to do if vm on proxmox.
If I do on bare metal somehow able to install but NW frozen and not able to do ssh host as well. Then if I remove /etc/openvswitch folder and restart the node then I can login at least but ovnkube-node pod not coming up.
https://github.com/ovn-kubernetes/ovn-kubernetes/issues/5091#issuecomment-2774112661
@garca-cloud Thanks for updates.
I did with flavio' helm but not able to do if vm on proxmox.
If I do on bare metal somehow able to install but NW frozen and not able to do ssh host as well. Then if I remove /etc/openvswitch folder and restart the node then I can login at least but ovnkube-node pod not coming up.
About the networking issue, like mentionated in the report #5091, at this moment before shutdown the vm on proxmox lab we deleted the folder /etc/origin/openvswitch. When the vm rebooted the network working fine, we only add manually the dns on the "br-interface" to permit to resolve the domains on coredns pod. BTW I think we'll open a bug report too or we can think to automatize these steps.
@garca-cloud Forgot to update ..
If I somehow make all pod running state in ovn-kubernetes namespace ... but it throwing error for other pod creations.
oc get po -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system cloud-controller-manager-rke2-master-01 1/1 Running 9 (5m18s ago) 87m
kube-system etcd-rke2-master-01 1/1 Running 6 87m
kube-system helm-install-rke2-ingress-nginx-vjw5m 0/1 ContainerCreating 0 87m
kube-system helm-install-rke2-metrics-server-rw79m 0/1 ContainerCreating 0 87m
kube-system helm-install-rke2-runtimeclasses-gks7c 0/1 ContainerCreating 0 87m
kube-system kube-apiserver-rke2-master-01 1/1 Running 4 87m
kube-system kube-controller-manager-rke2-master-01 1/1 Running 9 (5m18s ago) 87m
kube-system kube-scheduler-rke2-master-01 1/1 Running 5 (5m22s ago) 87m
kube-system rke2-coredns-rke2-coredns-6d54d78648-5hm88 0/1 ContainerCreating 0 87m
kube-system rke2-coredns-rke2-coredns-autoscaler-596dcdf688-9t9c8 0/1 ContainerCreating 0 87m
ovn-kubernetes ovnkube-db-67885fd9c4-4vgp6 2/2 Running 0 3m5s
ovn-kubernetes ovnkube-master-849c99f446-fj6nb 2/2 Running 0 3m5s
ovn-kubernetes ovnkube-node-vwqbb 3/3 Running 0 3m5s
ovn-kubernetes ovs-node-f9l6z 1/1 Running 0 3m5s
k describe po rke2-coredns-rke2-coredns-6d54d78648-5hm88 -n kube-system
Name: rke2-coredns-rke2-coredns-6d54d78648-5hm88
Namespace: kube-system
Priority: 2000000000
Priority Class Name: system-cluster-critical
Service Account: coredns
Node: rke2-master-01/192.168.0.124
Start Time: Tue, 08 Apr 2025 17:38:00 +0000
Labels: app.kubernetes.io/instance=rke2-coredns
app.kubernetes.io/name=rke2-coredns
k8s-app=kube-dns
pod-template-hash=6d54d78648
Annotations: checksum/config: 2792e765032b7469751649345702e3c674bb366ff865f526b19d92f7c39a2b4e
k8s.ovn.org/pod-networks:
{"default":{"ip_addresses":["10.244.0.6/24"],"mac_address":"0a:58:0a:f4:00:06","gateway_ips":["10.244.0.1"],"routes":[{"dest":"10.244.0.0/...
Status: Pending
IP:
IPs: <none>
Controlled By: ReplicaSet/rke2-coredns-rke2-coredns-6d54d78648
Containers:
coredns:
Container ID:
Image: rancher/hardened-coredns:v1.12.0-build20241126
Image ID:
Ports: 53/UDP, 53/TCP, 9153/TCP
Host Ports: 0/UDP, 0/TCP, 0/TCP
Args:
-conf
/etc/coredns/Corefile
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Limits:
cpu: 100m
memory: 128Mi
Requests:
cpu: 100m
memory: 128Mi
Liveness: http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5
Readiness: http-get http://:8181/ready delay=30s timeout=5s period=10s #success=1 #failure=5
Environment: <none>
Mounts:
/etc/coredns from config-volume (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jjvrj (ro)
Conditions:
Type Status
PodReadyToStartContainers False
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
config-volume:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: rke2-coredns-rke2-coredns
Optional: false
kube-api-access-jjvrj:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Guaranteed
Node-Selectors: kubernetes.io/os=linux
Tolerations: CriticalAddonsOnly op=Exists
node-role.kubernetes.io/control-plane:NoSchedule op=Exists
node-role.kubernetes.io/etcd:NoExecute op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 87m default-scheduler 0/1 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/not-ready: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
Warning FailedScheduling 77m (x2 over 82m) default-scheduler 0/1 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/not-ready: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
Warning FailedScheduling 56m (x2 over 61m) default-scheduler 0/1 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/not-ready: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
Warning FailedScheduling 61m default-scheduler 0/1 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/not-ready: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
Warning FailedScheduling 24m (x4 over 40m) default-scheduler 0/1 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/not-ready: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
Warning FailedScheduling 11m (x3 over 21m) default-scheduler 0/1 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/not-ready: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
Warning FailedScheduling 5m26s default-scheduler 0/1 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/not-ready: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
Normal Scheduled 3m7s default-scheduler Successfully assigned kube-system/rke2-coredns-rke2-coredns-6d54d78648-5hm88 to rke2-master-01
Warning FailedCreatePodSandBox 3m6s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "195a46fb22447d83c4c2a6e31d4b05b8e8bf29c62548b8b46725a4c7cbb3f838": plugin type="loopback" failed (add): failed to find plugin "loopback" in path [/opt/cni/bin]
Warning FailedCreatePodSandBox 3m5s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "00ac31db5b8d58e95c083bc52a06375f0a46417f42a0dd14520a52887fe7e653": plugin type="loopback" failed (add): failed to find plugin "loopback" in path [/opt/cni/bin]
Warning FailedCreatePodSandBox 2m51s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "d0a974ecf671ede3570a79791ab991ca61e6f1e28a5ac201230646c24ede1e5f": plugin type="loopback" failed (add): failed to find plugin "loopback" in path [/opt/cni/bin]
Warning FailedCreatePodSandBox 2m39s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "eaf8495f3f5e4280375cc34b44944c442ac4cc7d413185a64e82fab1c43a6da7": plugin type="loopback" failed (add): failed to find plugin "loopback" in path [/opt/cni/bin]
Warning FailedCreatePodSandBox 2m27s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "751e4b2ff346b4cf3526c6747baf0d4d978a14bd16daf4607df8d6c8799f085d": plugin type="loopback" failed (add): failed to find plugin "loopback" in path [/opt/cni/bin]
Warning FailedCreatePodSandBox 2m14s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "0276961046ef259dce86496ec2eeb175bec10e10e1e24c386f147acf39ba9f22": plugin type="loopback" failed (add): failed to find plugin "loopback" in path [/opt/cni/bin]
Warning FailedCreatePodSandBox 2m1s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "c88251424f4ace1b277af2ee93b71be1f450161e29434abfc1269f31dcea5652": plugin type="loopback" failed (add): failed to find plugin "loopback" in path [/opt/cni/bin]
Warning FailedCreatePodSandBox 106s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "c5c206d20c7616f7dc3919224852345ee2ba8858de411ec0ab3a72beb62a8f99": plugin type="loopback" failed (add): failed to find plugin "loopback" in path [/opt/cni/bin]
Warning FailedCreatePodSandBox 94s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "ca0463b93ebfe770c2ad6351979e4b436f165b486442671bad7463527a006683": plugin type="loopback" failed (add): failed to find plugin "loopback" in path [/opt/cni/bin]
Warning FailedCreatePodSandBox 2s (x7 over 81s) kubelet (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "ac69253e2d6d7a467efc690a9239a7ef2f3f5627a4a07d27bfd2aa244aaaa2b5": plugin type="loopback" failed (add): failed to find plugin "loopback" in path [/opt/cni/bin]
Hi @cloudcafetech, the log is clear you need to install plugin loopback:
failed to find plugin "loopback" in path [/opt/cni/bin]
But it should be take by cni (ovn-kubernetes) https://github.com/rancher/rke2/issues/7981
This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 5 days.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
Hi! I wanted to follow up and see if you have had the opportunity to review this issue. It will be automatically closed in 5 days due to inactivity.
But it should be take by cni (ovn-kubernetes) rancher/rke2#7981
@cloudcafetech how was the cluster set up?
It is surprising to see the loopback CNI as we are not configuring it as part of ovn-kubernetes.
Please give the recently released version a try https://github.com/ovn-kubernetes/ovn-kubernetes/releases/tag/v1.1.0
using https://github.com/ovn-kubernetes/ovn-kubernetes/blob/master/helm/basic-deploy.sh should get you up and running quickly.
I am seeing something similar to the original issue; i.e. VM not getting a L3 UDN IP and wondering if the cause was determined for that..
I am able to use a L2 UDN and the kubevirt VM comes fine with the IP, but when i try a L3 UDN, i don't see the IP in the VM (or in the cloud-init in the VM). Would appreciated some inputs on this.
Environment: ovn-k8s is lagging upstream head (will update how close it is to head) k8s version: v1.31.9 kubevirt: v1.6.0 multus v4.2.2
Kubevirt patched with: kubectl -n kubevirt patch kubevirt kubevirt --type=json --patch '[{"op":"add","path":"/spec/configuration/network","value":{}},{"op":"add","path":"/spec/configuration/network/binding","value":{"l2bridge":{"domainAttachmentType":"managedTap","migration":{}}}}]'
UDN ns: udnprimary (with label: "k8s.ovn.org/primary-user-defined-network: """)
kubevirt spec
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
name: test-udnprimary-vm
namespace: udnprimary
spec:
runStrategy: Always
template:
metadata:
labels:
kubevirt.io/domain: test-udnprimary-vm
spec:
domain:
devices:
disks:
- name: rootdisk
disk:
bus: virtio
- name: cloudinitdisk
disk:
bus: virtio
interfaces:
- binding:
name: l2bridge
name: default
resources:
requests:
memory: 1Gi
networks:
- name: default
pod: {}
nodeSelector:
kubernetes.io/hostname:
====
What works: L2UDN
apiVersion: k8s.ovn.org/v1 kind: UserDefinedNetwork metadata: name: l2primary namespace: udnprimary spec: topology: Layer2 layer2: role: Primary subnets: - "XYZ"
kubevirt VM comes up with IP from the subnet.
what doesn't work (with the L2 UDN deleted) : L3UDN
apiVersion: k8s.ovn.org/v1
kind: UserDefinedNetwork
metadata:
name: l3primary
namespace: udnprimary
spec:
topology: Layer3
layer3:
role: Primary
subnets:
- cidr: <some cidr/mask>
hostSubnet:
The VM doesn't come up with IP; don't see cloud-init with the IP in the VM.
I see the IP has been assigned
k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["pod IP"],"mac_address":"pod MAC","routes":[{"dest":"...","nextHop":"..."},{"dest":"100.64.0.0/16","nextHop":"..."}],"mtu":"1440","ip_address":"...","role":"in
frastructure-locked"},"udnprimary/l3primary":{"ip_addresses":["..."],"mac_address":"...","gateway_ips":["10.255.246.1"],"routes":[{"dest":"...","nextHop":"..."},{"dest":"...","nextHop":"..."},{"dest":"100.65.0.0/16"
,"nextHop":"..."}],"mtu":"1440","ip_address":"...","gateway_ip":"...","role":"primary"}}'
k8s.v1.cni.cncf.io/network-status: |-
[{
"name": "default/ovn-primary",
"interface": "eth0",
"ips": [
"
but nothing in the vm
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host noprefixroute valid_lft forever preferred_lft forever 2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc fq_codel state UP group default qlen 1000 link/ether ...brd ff:ff:ff:ff:ff:ff altname enx0a580afff603 inet6 ...64 scope link noprefixroute valid_lft forever preferred_lft forever
the mac for enp1s0 is the right one from the pod annotation.
I suspect something is missing from the kubevirt side? I also tried with passt as: export passt_binding_image="quay.io/kubevirt/network-passt-binding:v1.6.0"
kubectl -n kubevirt patch kubevirt kubevirt --type=json --patch '[{"op":"add","path":"/spec/configuration/network","value":{}},{"op":"add","path":"/spec/configuration/network/binding","value":{"passt":{"computeResourceOverhead":{"requests":{"memory":"500Mi"}},"migration":{"method":"link-refresh"},"networkAttachmentDefinition":"udnprimary/l3primary","sidecarImage":"'"${passt_binding_image}"'"},"l2bridge":{"domainAttachmentType":"managedTap","migration":{}}}}]'
but same result. I looked at the e2e tests and see kubevirt uses L2 or localnet; some of the examples i found (including the workshop) was with L2. If there is an example with L3 and kubevirt, would appreciate a pointer.
Please let me know if i am missing something obvious or there were any recent push related to L3 UDN and kubevirt.
If you need further info., please let me know. thanks!
I am seeing something similar to the original issue; i.e. VM not getting a L3 UDN IP and wondering if the cause was determined for that..
I am able to use a L2 UDN and the kubevirt VM comes fine with the IP, but when i try a L3 UDN, i don't see the IP in the VM (or in the cloud-init in the VM). Would appreciated some inputs on this.
Environment: ovn-k8s is lagging upstream head (will update how close it is to head) k8s version: v1.31.9 kubevirt: v1.6.0 multus v4.2.2
Kubevirt patched with: kubectl -n kubevirt patch kubevirt kubevirt --type=json --patch '[{"op":"add","path":"/spec/configuration/network","value":{}},{"op":"add","path":"/spec/configuration/network/binding","value":{"l2bridge":{"domainAttachmentType":"managedTap","migration":{}}}}]'
UDN ns: udnprimary (with label: "k8s.ovn.org/primary-user-defined-network: """)
kubevirt spec
apiVersion: kubevirt.io/v1 kind: VirtualMachine metadata: name: test-udnprimary-vm namespace: udnprimary spec: runStrategy: Always template: metadata: labels: kubevirt.io/domain: test-udnprimary-vm spec: domain: devices: disks: - name: rootdisk disk: bus: virtio - name: cloudinitdisk disk: bus: virtio interfaces: - binding: name: l2bridge name: default resources: requests: memory: 1Gi networks: - name: default pod: {} nodeSelector: kubernetes.io/hostname: volumes: - name: rootdisk containerDisk: image: quay.io/containerdisks/fedora:latest - name: cloudinitdisk cloudInitNoCloud: userData: | #cloud-config user: fedora password: chpasswd: { expire: False } ssh_pwauth: True
====
What works: L2UDN
apiVersion: k8s.ovn.org/v1 kind: UserDefinedNetwork metadata: name: l2primary namespace: udnprimary spec: topology: Layer2 layer2: role: Primary subnets: - "XYZ"
kubevirt VM comes up with IP from the subnet.
what doesn't work (with the L2 UDN deleted) : L3UDN
apiVersion: k8s.ovn.org/v1 kind: UserDefinedNetwork metadata: name: l3primary namespace: udnprimary spec: topology: Layer3 layer3: role: Primary subnets: - cidr: <some cidr/mask> hostSubnet:
The VM doesn't come up with IP; don't see cloud-init with the IP in the VM.
I see the IP has been assigned
k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["pod IP"],"mac_address":"pod MAC","routes":[{"dest":"...","nextHop":"..."},{"dest":"100.64.0.0/16","nextHop":"..."}],"mtu":"1440","ip_address":"...","role":"in frastructure-locked"},"udnprimary/l3primary":{"ip_addresses":["..."],"mac_address":"...","gateway_ips":["10.255.246.1"],"routes":[{"dest":"...","nextHop":"..."},{"dest":"...","nextHop":"..."},{"dest":"100.65.0.0/16" ,"nextHop":"..."}],"mtu":"1440","ip_address":"...","gateway_ip":"...","role":"primary"}}' k8s.v1.cni.cncf.io/network-status: |- [{ "name": "default/ovn-primary", "interface": "eth0", "ips": [ "" ], "mac": "xx:xx:xx:xx:xx:xx", "dns": {} },{ "name": "default/ovn-primary", "interface": "ovn-udn1", "ips": [ "" ], "mac": "yy:yy:yy:yy:yy:yy", "default": true, "dns": {} }]
but nothing in the vm
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host noprefixroute valid_lft forever preferred_lft forever 2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc fq_codel state UP group default qlen 1000 link/ether ...brd ff:ff:ff:ff:ff:ff altname enx0a580afff603 inet6 ...64 scope link noprefixroute valid_lft forever preferred_lft forever
the mac for enp1s0 is the right one from the pod annotation.
I suspect something is missing from the kubevirt side? I also tried with passt as: export passt_binding_image="quay.io/kubevirt/network-passt-binding:v1.6.0"
kubectl -n kubevirt patch kubevirt kubevirt --type=json --patch '[{"op":"add","path":"/spec/configuration/network","value":{}},{"op":"add","path":"/spec/configuration/network/binding","value":{"passt":{"computeResourceOverhead":{"requests":{"memory":"500Mi"}},"migration":{"method":"link-refresh"},"networkAttachmentDefinition":"udnprimary/l3primary","sidecarImage":"'"${passt_binding_image}"'"},"l2bridge":{"domainAttachmentType":"managedTap","migration":{}}}}]'
but same result. I looked at the e2e tests and see kubevirt uses L2 or localnet; some of the examples i found (including the workshop) was with L2. If there is an example with L3 and kubevirt, would appreciate a pointer.
Please let me know if i am missing something obvious or there were any recent push related to L3 UDN and kubevirt.
If you need further info., please let me know. thanks!
We don't support VMs + layer3, so there is no DHCP server going on with layer3
https://github.com/ovn-kubernetes/ovn-kubernetes/blob/56d14a3074ba7df00a6b68dfe4ac60aff7fc624a/go-controller/pkg/ovn/base_network_controller_user_defined.go#L945
This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 5 days.
This issue was closed because it has been stalled for 5 days with no activity.