Are multiple vms accessing single gvproxy supported?

[nirs]

When trying to use multiple vfkit vms, only the first one seems to be connected to gvproxy, and the second one start very slow and finally do not get an ip address.

I see that podman is using gvproxy socket per machine, and it supports only one running machine, so I guess the answer is no. This should probably be documented.

Shell 1:

% cat start-gvproxy-for-vfkit.sh 
rm -f /tmp/vfkit.sock
gvproxy \
    --listen unix:///tmp/network.sock \
    --listen-vfkit unixgram:///tmp/vfkit.sock

% sh start-gvproxy-for-vfkit.sh 
INFO[0000] gvproxy version v0.7.4                       
INFO[0000] waiting for clients...                       
INFO[0000] listening unix:///tmp/network.sock           
INFO[0008] new connection from /Users/nsoffer/Library/Application Support/vfkit/net-79779-3390717601.sock to /tmp/vfkit.sock 

Shell 2 (got ip address):

% sh start-vfkit.sh
INFO[0000] &{2 2048    {[efi variable-store=efi-variable-store create] true}  [virtio-blk,path=disk.img usb-mass-storage,path=seed.iso virtio-serial,stdio virtio-net,unixSocketPath=/tmp/vfkit.sock,mac=5a:94:ef:e4:0c:02] none://  false} 
INFO[0000] boot parameters: &{EFIVariableStorePath:efi-variable-store CreateVariableStore:true} 
INFO[0000]                                              
INFO[0000] virtual machine parameters:                  
INFO[0000] 	vCPUs: 2                                    
INFO[0000] 	memory: 2048 MiB                            
INFO[0000]                                              
INFO[0000] Adding virtio-blk device (imagePath: disk.img) 
INFO[0000] Adding USB mass storage device (imagePath: seed.iso) 
INFO[0000] Adding stdio console                         
INFO[0000] Adding virtio-net device (nat: false macAddress: [5a:94:ef:e4:0c:02]) 
INFO[0000] Using unix socket /tmp/vfkit.sock            
INFO[0000] local: /Users/nsoffer/Library/Application Support/vfkit/net-79779-3390717601.sock remote: /tmp/vfkit.sock 
INFO[0000] virtual machine is running                   
INFO[0000] waiting for VM to stop                       

Fedora Linux 40 (Cloud Edition)
Kernel 6.8.5-301.fc40.aarch64 on an aarch64 (hvc0)

eth0: 192.168.127.3 fe80::5894:efff:fee4:c02
...

$ ip addr show dev eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 5a:94:ef:e4:0c:02 brd ff:ff:ff:ff:ff:ff
    altname enp0s1
    inet 192.168.127.3/24 brd 192.168.127.255 scope global dynamic noprefixroute eth0
       valid_lft 2752sec preferred_lft 2752sec
    inet6 fe80::5894:efff:fee4:c02/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

Shell 3 (no ip address):

 % sh start-vfkit.sh
INFO[0000] &{2 2048    {[efi variable-store=efi-variable-store create] true}  [virtio-blk,path=disk.img usb-mass-storage,path=seed.iso virtio-serial,stdio virtio-net,unixSocketPath=/tmp/vfkit.sock,mac=5a:94:ef:e4:0c:01] none://  false} 
INFO[0000] boot parameters: &{EFIVariableStorePath:efi-variable-store CreateVariableStore:true} 
INFO[0000]                                              
INFO[0000] virtual machine parameters:                  
INFO[0000] 	vCPUs: 2                                    
INFO[0000] 	memory: 2048 MiB                            
INFO[0000]                                              
INFO[0000] Adding virtio-blk device (imagePath: disk.img) 
INFO[0000] Adding USB mass storage device (imagePath: seed.iso) 
INFO[0000] Adding stdio console                         
INFO[0000] Adding virtio-net device (nat: false macAddress: [5a:94:ef:e4:0c:01]) 
INFO[0000] Using unix socket /tmp/vfkit.sock            
INFO[0000] local: /Users/nsoffer/Library/Application Support/vfkit/net-79782-3629316716.sock remote: /tmp/vfkit.sock 
INFO[0000] virtual machine is running                   
INFO[0000] waiting for VM to stop                       

Fedora Linux 40 (Cloud Edition)
Kernel 6.10.4-200.fc40.aarch64 on an aarch64 (hvc0)
...

$ ip addr show dev eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 5a:94:ef:e4:0c:01 brd ff:ff:ff:ff:ff:ff
    altname enp0s1

[balajiv113]

@nirs In general, gvisor-tap-vsock supports multiple vms accessing via single gvproxy. Lima supports and uses it for lima:user-v2

This has became a limitation mainly because of how code written in https://github.com/containers/gvisor-tap-vsock/blob/main/cmd/gvproxy/main.go#L422 As per the code, it will listen only for one connection.

[nirs]

@balajiv113 Yes it is clear now why it does not work.

lima user-v2 does not provide shared network, it only allows access from VM to VM, but there is no way to access the VMs from the host using the VM IP address.

[cfergeau]

As per the code, it will listen only for one connection.

vfkit followed the same pattern as the other listeners (qemu, bess, ...). Only vpnkit/hyperkit loops for some reason. PRs adding a loop there are welcome if this works as expected!

[MrEcco]

@cfergeau @balajiv113 @nirs Hey guys, I have fixed the issue with qemu/vfkit/bes/etc listeners in this PR: https://github.com/containers/gvisor-tap-vsock/pull/484. Can you please support it with 👍🏻?

I tested it on arm64 MacOS, with libvirt+qemu. Both VMs have access to internet and see each other. I think it's pretty close to shared network. TCP forwarding feature is also available (now with simple configuration file).

[nirs]

@cfergeau @balajiv113 @nirs Hey guys, I have fixed the issue with qemu/vfkit/bes/etc listeners in this PR: #484. Can you please support it with 👍🏻?

@MrEcco If you are using vfkit, krunkit, or qemu, you should be able to use a shared network with vmnet-helper.