Libvirt driver, for hypervisors on remote machines

[afbjorklund]

Description

You can use libvirt for talking to the local libvirtd, but it can also open a remote connection:

https://libvirt.org/remote.html

This makes it interesting for starting the QEMU machine on a local server, instead of localhost.

There is also the qemu://session, but it doesn't really add anything over running qemu-system?

---

Thinking of adding a new "virt" driver, for using libvirt. The goal is to run qemu remotely.

• qemu:///system connects to a system mode daemon. (like rootful)

• qemu:///session connects to a session mode daemon. (like rootless)

The libvirt libraries bring a lot of dependencies though, so hide them in an optional .so plugin...

• https://github.com/lima-vm/lima/pull/2000

• https://github.com/lima-vm/lima/discussions/2007

---

Alternatives

The alternative (to qemu+ssh://) is to ssh to the remote host, and run limactl from there...

https://libvirt.org/uri.html

The libvirt driver would be more interesting when coupled with oVirt, to allocate a VM ?

But that would probably be better as a separate "cloud" driver, using the go-ovirt-client

[afbjorklund]

Initial PR:

• #2032

Mostly to test the plugin system, it can successfully call the libvirt API without adding to limactl.

INFO[0000] Starting the instance "virt" with VM driver "virt" 
INFO[0000] VERSION: 6000000                             
INFO[0000] Version: 6000000              

29M	_output/bin/limactl
4,7M	_output/lib/lima/plugin/virt.so

$ ldd _output/bin/limactl 
	linux-vdso.so.1 (0x00007fff965a5000)
	libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f9fc6270000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f9fc624d000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f9fc6247000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f9fc6055000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f9fc62b3000)
$ ldd _output/lib/lima/plugin/virt.so 
	linux-vdso.so.1 (0x00007ffd183dd000)
	libvirt-lxc.so.0 => /lib/x86_64-linux-gnu/libvirt-lxc.so.0 (0x00007f96c9eb8000)
	libvirt-qemu.so.0 => /lib/x86_64-linux-gnu/libvirt-qemu.so.0 (0x00007f96c9eb3000)
	libvirt.so.0 => /lib/x86_64-linux-gnu/libvirt.so.0 (0x00007f96c9a5d000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f96c9a3a000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f96c9848000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f96c96f9000)
	libcap-ng.so.0 => /lib/x86_64-linux-gnu/libcap-ng.so.0 (0x00007f96c96ef000)
	libyajl.so.2 => /lib/x86_64-linux-gnu/libyajl.so.2 (0x00007f96c96e3000)
	libnl-3.so.200 => /lib/x86_64-linux-gnu/libnl-3.so.200 (0x00007f96c96c0000)
	libaudit.so.1 => /lib/x86_64-linux-gnu/libaudit.so.1 (0x00007f96c9694000)
	libdevmapper.so.1.02.1 => /lib/x86_64-linux-gnu/libdevmapper.so.1.02.1 (0x00007f96c9629000)
	libnuma.so.1 => /lib/x86_64-linux-gnu/libnuma.so.1 (0x00007f96c961c000)
	libacl.so.1 => /lib/x86_64-linux-gnu/libacl.so.1 (0x00007f96c960f000)
	libxml2.so.2 => /lib/x86_64-linux-gnu/libxml2.so.2 (0x00007f96c9455000)
	libgio-2.0.so.0 => /lib/x86_64-linux-gnu/libgio-2.0.so.0 (0x00007f96c9274000)
	libgobject-2.0.so.0 => /lib/x86_64-linux-gnu/libgobject-2.0.so.0 (0x00007f96c9214000)
	libglib-2.0.so.0 => /lib/x86_64-linux-gnu/libglib-2.0.so.0 (0x00007f96c90ea000)
	libsasl2.so.2 => /lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007f96c90cd000)
	libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x00007f96c90a0000)
	libapparmor.so.1 => /lib/x86_64-linux-gnu/libapparmor.so.1 (0x00007f96c908b000)
	libdbus-1.so.3 => /lib/x86_64-linux-gnu/libdbus-1.so.3 (0x00007f96c903a000)
	libgnutls.so.30 => /lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007f96c8e65000)
	libcurl-gnutls.so.4 => /lib/x86_64-linux-gnu/libcurl-gnutls.so.4 (0x00007f96c8dd5000)
	libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f96c8dd0000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f96c8dc8000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f96c8dad000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f96ca3c4000)
	libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007f96c8d80000)
	libicuuc.so.66 => /lib/x86_64-linux-gnu/libicuuc.so.66 (0x00007f96c8b9a000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f96c8b7e000)
	liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f96c8b53000)
	libgmodule-2.0.so.0 => /lib/x86_64-linux-gnu/libgmodule-2.0.so.0 (0x00007f96c8b4d000)
	libmount.so.1 => /lib/x86_64-linux-gnu/libmount.so.1 (0x00007f96c8aed000)
	libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f96c8ad1000)
	libffi.so.7 => /lib/x86_64-linux-gnu/libffi.so.7 (0x00007f96c8ac5000)
	libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007f96c8a52000)
	libpcre2-8.so.0 => /lib/x86_64-linux-gnu/libpcre2-8.so.0 (0x00007f96c89bf000)
	libsystemd.so.0 => /lib/x86_64-linux-gnu/libsystemd.so.0 (0x00007f96c8910000)
	libp11-kit.so.0 => /lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007f96c87da000)
	libidn2.so.0 => /lib/x86_64-linux-gnu/libidn2.so.0 (0x00007f96c87b9000)
	libunistring.so.2 => /lib/x86_64-linux-gnu/libunistring.so.2 (0x00007f96c8637000)
	libtasn1.so.6 => /lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007f96c8621000)
	libnettle.so.7 => /lib/x86_64-linux-gnu/libnettle.so.7 (0x00007f96c85e5000)
	libhogweed.so.5 => /lib/x86_64-linux-gnu/libhogweed.so.5 (0x00007f96c85ae000)
	libgmp.so.10 => /lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f96c852a000)
	libnghttp2.so.14 => /lib/x86_64-linux-gnu/libnghttp2.so.14 (0x00007f96c8501000)
	librtmp.so.1 => /lib/x86_64-linux-gnu/librtmp.so.1 (0x00007f96c84e1000)
	libssh.so.4 => /lib/x86_64-linux-gnu/libssh.so.4 (0x00007f96c8473000)
	libpsl.so.5 => /lib/x86_64-linux-gnu/libpsl.so.5 (0x00007f96c845e000)
	libgssapi_krb5.so.2 => /lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007f96c8411000)
	libldap_r-2.4.so.2 => /lib/x86_64-linux-gnu/libldap_r-2.4.so.2 (0x00007f96c83bb000)
	liblber-2.4.so.2 => /lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007f96c83aa000)
	libbrotlidec.so.1 => /lib/x86_64-linux-gnu/libbrotlidec.so.1 (0x00007f96c839c000)
	libicudata.so.66 => /lib/x86_64-linux-gnu/libicudata.so.66 (0x00007f96c68d9000)
	libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f96c66bf000)
	libblkid.so.1 => /lib/x86_64-linux-gnu/libblkid.so.1 (0x00007f96c6668000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f96c665e000)
	liblz4.so.1 => /lib/x86_64-linux-gnu/liblz4.so.1 (0x00007f96c663d000)
	libgcrypt.so.20 => /lib/x86_64-linux-gnu/libgcrypt.so.20 (0x00007f96c651d000)
	libcrypto.so.1.1 => /lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f96c6247000)
	libkrb5.so.3 => /lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f96c616a000)
	libk5crypto.so.3 => /lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007f96c6139000)
	libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f96c6132000)
	libkrb5support.so.0 => /lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007f96c6121000)
	libgssapi.so.3 => /lib/x86_64-linux-gnu/libgssapi.so.3 (0x00007f96c60dc000)
	libbrotlicommon.so.1 => /lib/x86_64-linux-gnu/libbrotlicommon.so.1 (0x00007f96c60b9000)
	libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007f96c6096000)
	libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007f96c608f000)
	libheimntlm.so.0 => /lib/x86_64-linux-gnu/libheimntlm.so.0 (0x00007f96c6081000)
	libkrb5.so.26 => /lib/x86_64-linux-gnu/libkrb5.so.26 (0x00007f96c5fee000)
	libasn1.so.8 => /lib/x86_64-linux-gnu/libasn1.so.8 (0x00007f96c5f48000)
	libhcrypto.so.4 => /lib/x86_64-linux-gnu/libhcrypto.so.4 (0x00007f96c5f10000)
	libroken.so.18 => /lib/x86_64-linux-gnu/libroken.so.18 (0x00007f96c5ef7000)
	libwind.so.0 => /lib/x86_64-linux-gnu/libwind.so.0 (0x00007f96c5ecd000)
	libheimbase.so.1 => /lib/x86_64-linux-gnu/libheimbase.so.1 (0x00007f96c5eb9000)
	libhx509.so.5 => /lib/x86_64-linux-gnu/libhx509.so.5 (0x00007f96c5e6b000)
	libsqlite3.so.0 => /lib/x86_64-linux-gnu/libsqlite3.so.0 (0x00007f96c5d42000)
	libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007f96c5d07000)

[afbjorklund]

LIBVIRT_DEFAULT_URI=qemu:///session

Adding support for qemu-session is useful for development and debugging:

But it uses a lot of qemu-specific hacks, rather than the actual libvirt API...

LIBVIRT_DEFAULT_URI=qemu:///system

The qemu-system is more "real", but it (libvirtd) cannot access ~/.lima directly.

So that means that all images have to be copied to /var/lib/libvirt/images , etc.

https://libvirt.org/storage.html

[afbjorklund]

The actual implementation is mostly an exercise in XML templating. With some API exercises.

https://libvirt.org/formatdomain.html

https://libvirt.org/formatnetwork.html

https://pkg.go.dev/libvirt.org/go/libvirt

[vsoch]

@afbjorklund I have a (possibly dumb) question. If lima can support remote machines, could it support connecting more than one remote machine? E.g., bringing up VMs on different hypervisor hosts and then networking them all?

And actually, what if I just brought up llima VMs on separate physical machines, can I expose their ip addresses in some way to connect and have a cluster?

[afbjorklund]

I suppose it is possible, but it sounds like a job that would be better for norouter.io or similar dedicated software?

• https://norouter.io/docs/getting-started/first-example/

The out-of-the-box experience is that you get one SSH connection to each VM, everything beyond that is "extra"...

And it seems oVirt is deprecated ? (For KubeVirt)

[vsoch]

For context, I want to get the same setup (usernetes with flux and VMs) on a small multi-node cluster. The colleague that is helping me setup has been using ovirt with ansible, and instead of starting fresh figuring that out I was wondering if Lima could still be used.

should we try KubeVirt instead for this use case? Still with ansible or something else?

I greatly appreciate your insights, still fairly new to automating setup of VMs beyond APIs provided by cloud builders and Vagrant.

[afbjorklund]

I should have mentioned that both oVirt and KubeVirt are out of scope for this issue, about libvirt support...

[vsoch]

I thought it might be related to remote machines, so I chose it.

[afbjorklund]

The main difference is that they (oVirt/KubeVirt providers) will choose a VM host for you, from "the cloud"...

With this libvirt driver, you will specify an URI like:

uri_aliases = [
  "hail=qemu+ssh://[email protected]/system",
  "sleet=qemu+ssh://[email protected]/system",
]

And it would talk to one machine at a time ($LIBVIRT_DEFAULT_URI).

So the workaround is that we ssh to the server, and talk to qemu-system on it (similar to qemu:///session)

ssh [email protected] limactl ...

But then it would be a different lima list, there.

It would not blend in with the local instances.

[afbjorklund]

Will do another attempt of PoC implementation, but using virsh exec instead of libvirt pkg.