`kvm` support?

[dhess]

trafficstars

Hi, thanks for creating this project! I'm pretty dissatisfied with the price/performance of dedicated aarch64 hardware compared to Apple Silicon, so I'm considering using one of our low-end M1 Macs to replace an aarch64-linux machine running NixOS.

What's the status of kvm support in M1 NixOS (as in nix.settings.system-features = [ "kvm" ...])? I'd like to be able to run NixOS tests and nixpkgs builders like pkgs.vmTools.runInLinuxVM that rely on this feature.

Based on some quick research, it sounds like there might be some issues with KVM support in QEMU on Asahi Linux, and maybe more specifically on vanilla M1 CPUs (as opposed to M1 Pro, Max, & Ultra), but most of the issues I've seen are with Windows, rather than just virtualizing NixOS on NixOS.

[tpwrules]

I tried building NixOS tests on the latest release and my M1 Max machine and there's definitely something wrong here. /dev/kvm is present, but the QEMU process seems to just hang at 100% CPU so something is completely broken with KVM and/or the software emulation is excruciatingly slow.

Would be interested to here from other Asahi users. I don't have time to debug this at this moment.

[dhess]

Thanks for testing it out. That's disappointing to hear that it doesn't work well on an M1 Max, which I'd been led to believe worked fine (per this comment).

Supposedly the problems have to do with QEMU not being able to deal with heterogenous CPUs (efficiency vs performance cores), and if you can manage to create a taskset/cpuset that contains only the peformance cores and run QEMU on that, it works.

[uuuvn]

I use virtualization heavily on my m1 max and i noticed only two problems: 1). https://www.reddit.com/r/AsahiLinux/comments/134g00u/strange_scheduling_latency_on_the_host_when_kvm/ Fixed in newer m1n1 version 2). Guest may fail to initialize cores if you don't pin them to only one type of cores. In libvirtd i just do <vcpu placement="static" cpuset="2-9">8</vcpu> (on m1 max efficency cores are 0-1 and the rest are performance ones)

[zzywysm]

~KVM works ok for me. I use virt-manager-qt to create and run VMs.~

I recommend adding your main user to the kvm group within configuration.nix.

virt-manager-qt doesn't add video, graphics, mouse, or keyboard by default, it seems, so you'll need to manually add that hardware to your VM. ~Expect the mouse experience to suck.~

EDIT: I fixed my remaining issues by using virt-manager instead of virt-manager-qt. If you're using a MacBook's trackpad, make sure you add a trackpad to the VM hardware, not a mouse. (You still need to add video, graphics, and a keyboard.). KVM works pretty well with these changes.