Multiple VMs Might Break Network Connections

[iwinux]

OS / App Versions

• macOS 13.5 (x86-64)
• QEMU 7.2.1
• socket_vmnet 1.1.2 built from source

Problem

The network of currently started QEMU VM stops working (symptom: ping 1.1.1.1 times out) when another VM (with unique MAC address) boots up but DOES NOT initiate any DHCP request.

However, after the second VM finishes its DHCP request, the network on first VM resumes to normal.

Steps to Reproduce

1. start socket_vmnet

sudo env DEBUG=true /usr/local/bin/socket_vmnet \
	--socket-group=admin \
	--vmnet-mode=shared \
	--vmnet-gateway=10.233.2.1 \
	--vmnet-dhcp-end=10.233.2.250 \
	--vmnet-mask=255.255.255.0 \
	/run/vmnet.sock

2. start VM `vmnet-test-01`

# Alpine Live ISOs available at https://www.alpinelinux.org/downloads/
# The "Virtual" ones are smallest

/usr/local/bin/socket_vmnet_client \
    /run/vmnet.sock \
    qemu-system-x86_64 \
        -name "vmnet-test-01" \
        -machine type=q35,accel=hvf \
        -cpu host \
        -smp cpus=1,sockets=1,cores=1,threads=1 \
        -m 1024 \
        -cdrom "$HOME/Downloads/alpine-virt-3.18.2-x86_64.iso" \
        -netdev socket,id=net0,fd=3 \
        -device "virtio-net-pci,netdev=net0,mac=de:ad:be:ef:00:01" \
        -object rng-random,filename=/dev/urandom,id=rng0 \
        -device virtio-rng-pci,rng=rng0 \
        -device virtio-gpu-pci \
        -nographic

3. acquire IP via DHCP for vmnet-test-01

# login with user "root"

cat > /etc/network/interfaces <<-END
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet dhcp
END

rc-service networking start

# it pings successfully
ping -c 5 -w 1 1.1.1.1

4. start VM `vmnet-test-02`

/usr/local/bin/socket_vmnet_client \
    /run/vmnet.sock \
    qemu-system-x86_64 \
        -name "vmnet-test-02" \
        -machine type=q35,accel=hvf \
        -cpu host \
        -smp cpus=1,sockets=1,cores=1,threads=1 \
        -m 1024 \
        -cdrom "$HOME/Downloads/alpine-virt-3.18.2-x86_64.iso" \
        -netdev socket,id=net0,fd=3 \
        -device "virtio-net-pci,netdev=net0,mac=de:ad:be:ef:00:02" \
        -object rng-random,filename=/dev/urandom,id=rng0 \
        -device virtio-rng-pci,rng=rng0 \
        -device virtio-gpu-pci \
        -nographic

5. after 1-2 minutes, network on vmnet-test-01 stops working

# timeout
ping -c 5 -w 1 1.1.1.1

# DHCP cannot renew lease
rc-service networking restart

6. repeat step 3 for vmnet-test-02

# login with user "root"

cat > /etc/network/interfaces <<-END
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet dhcp
END

rc-service networking start

# it pings successfully
ping -c 5 -w 1 1.1.1.1

7. surprisingly, now networks on both VMs resume to normal

Debug Log

https://gist.github.com/iwinux/305a9bea6b77c5c44494bb1883354f72

[nirs]

@iwinux can you reproduce this with lima >= 1.0.0?

Please test with current master, we have better logs now (see /var/log/socket_vmnet/stderr).

[nirs]

@iwinux can you reproduce this with lima >= 1.0.0?

I see you don't use lima, so it is not relevant.

[iwinux]

@nirs Cool, just noticed a lot of new commits since September. Will try with latest master version later.

[nirs]

socket_vmnet send packets received from one vm or the vmnet interface to all other vms. It uses writev() which will block forever if the other side does not read from the socket.

Maybe qemu is not reading from the socket until the guest set up the network. In this case the socket buffer will become full at some point. This will cause socket_vment to block trying to send packet to the new vm.

It may help to ask about this in qemu-discuss mailing list.

If this is the root cause, it will be fixed by #77.

Another way to avoid this is to implement send timeout and drop the packet.