lima-vm
Improve socket_vmnet performance using github.com/nirs/vmnet-helper
At KubeCon @nirs and I discussed improving the vmnet performance using https://github.com/nirs/vmnet-helper
Two options:
- Option 1: port vmnet-helper improvements to socket_vmnet
- Option 2: implement the support for vmnet-helper
Either way, the change will be almost transparent to users.
The network names will remain lima:shared, lima:bridged, and lima:host.
Option 1: port vmnet-helper improvements to socket_vmnet
This plan will port the performance improvements of vmnet-helper to socket_vmnet , preserving the current CLI of socket_vmnet.
Feasibility is yet to be investigated.
As this will incur a massive rewrite, it might be also a good chance to consider rewriting the entire code in Swift.
Pros:
- (Almost) no need to touch the code of Lima
- Less confusing to users
Cons:
- Massive change might be needed on socket_vmnet
Option 2: implement the support for vmnet-helper
networks.yaml will have a vmnetBackend: (vmnet-helper|socket_vmnet|auto) to switch the backend
until we can fully deprecate the support for socket_vmnet.
Pros:
- No need to touch the code of vmnet-helper
Cons:
- Brings more code complexity to Lima
- Confusing to have two backends with very similar names
cc @nirs Do you plan to work on this ?
Yes.
Regarding the propose solutions:
Option 1: port vmnet-helper improvements to socket_vmnet
I don't think this can work. vment-helper uses the process per vm model.
For every vm we have:
- helper process
- vmnet interface (not using share interface for all)
- the helper process drop privileges to the current user after starting the vmnet interface
- thread forwarding packets from the vm to vmnet
- thread forwarding packets from vment to the vm
- forwarding is done with sendmsg_x and recvmsg_x to copy multiple packets per system call
- vment/kernel is responsible for routing packets to the right destination
- since threads do not interact in any way, there is no locking
- all memory is allocated during start, no malloc/free per packet
- no need for managing daemon used my multipole instances (stop daemon when last instance stops)
- no need for launchd service
- using datagram socket - no length prefixed packet and multiple syscalls per packet
- can use the MAC address assigned by vmnet (Apple recommended usage). using MAC address generated by lima also works (tested in vment-helper CI).
To provide similar API to lima, we can create a daemon starting and stoping the helper process and passing it a fd from lima. I don't see much value in this, since we already manage process per vm, and running another process per vm is easy. The only advantage is avoiding sudo configuration, but since we already require sudo for lima:* this should not be an issue. If we want to do this, we can add a tiny daemon in Go accepting connected fd and starting a helper process with the fd.
Option 2: implement the support for vmnet-helper
This is my plan.
Cons: Brings more code complexity to Lima
I think this should not be a problem if we can integrate it cleanly.
Confusing to have two backends with very similar names
Should not be a big issue. If everything work well we can have 2 backends for one release and drop the older one in the next release.
We can have a vment package with 2 implementations, hiding the differences from the management code.
For reference see the minikube vmnet package:
- https://github.com/kubernetes/minikube/blob/master/pkg/drivers/common/vmnet/vmnet.go
Driver using either nat (same as VZNAT) or vmnet-helper https://github.com/kubernetes/minikube/blob/f7a9245a5ce7a2261ad1535c1a6bc91cdd9cbbd1/pkg/drivers/vfkit/vfkit.go#L237
I plan to start with quick proof of concept to see vz vm running with vmnet-helper. It will take few week until I have time to work on this.
I don't think this can work. vment-helper uses the process per vm model.
Is this really the main reason why vmnet-helper outperforms socket_vmnet?
Can we just achieve similar performance by modifying socket_vmnet to:
- Eliminate flooding
- Switch from SOCK_STREAM to SOCK_DGRAM
- Use sendmsg_x() and recvmsg_x()