graphical/vfio usecase

[yangm97]

Because gpu hot plugging between the host and guest vms is not always possible and given microvm allows for the guest nixOS to be very integrated with the host nixOS, this should be useful for single gpu passthrough/vfio use cases.

[astro]

I have added it to most hypervisors. Please give that a try and do leave some feedback.

[yangm97]

Wow, thank you!

I would suggest making this import overrideable, because it breaks pretty every graphical package:

https://github.com/astro/microvm.nix/blob/682b1e76e7fefdf350f3dc9e84002a8488e2b86d/nixos-modules/microvm/system.nix#L9

I also noticed this unit https://github.com/astro/microvm.nix/blob/5181933ca7bbaad37ceb82a848630ea3b30d522b/nixos-modules/host.nix#L153 does not set up user/group permissions for the device node, so the hypervisor fails to start i.e.

qemu-system-x86_64: -device vfio-pci,host=0000:02:00.0,multifunction=on: vfio 0000:02:00.0: failed to open /dev/vfio/51: Permission denied

[astro]

Right, I removed the minimal profile.

For the second issue, we need to get from the given PCI ID to the /dev/vfio/... path and apply permissions. Can you do a PR for this?

[yangm97]

For the time being I've done this wonderful piece of nix to deal with permissions 🤣

 systemd.services."microvm-pci-devices@".serviceConfig.ExecStartPost =
    let
      user = "microvm";
      group = "kvm";
      path = "/dev/vfio";
      permisssionsScript = pkgs.writeScript "permissions-setup" ''
        #! ${pkgs.runtimeShell} -e
        cd ${path}
        chown ${user}:${group} *
        chmod g+w *
      '';
    in
    "${permisssionsScript} %i";

Meanwhile I'm trying to figure out why qemu appears to be deadlocking (100% cpu usage on a single core) when I try to passtrough my RX560D...

[vilvo]

Right, I removed the minimal profile.

Could we make the profile configurable instead? With minimal as default. That would be profile of choice for the host. Guest could have other profiles.

I'd have similar need as @yangm97 - pass-through the GPU to guest but also move graphical packages there.

[astro]

Inclusion of a profile is not an overridable option. Therefore I would like to leave this to the user to pick the right one in their microvm config themself.

[vilvo]

Inclusion of a profile is not an overridable option. Therefore I would like to leave this to the user to pick the right one in their microvm config themself.

Sure, makes sense. I can import profile(s) in own configs.

[astro]

Can anyone confirm a working setup? Do you have practical hints that I may add to the docs?

[yangm97]

I got lost with the AMD passthrough issues but these don't appear to be microvm related... I know it is able to do passthroughs, used with libvirt in the past, but either something regressed with nix or I did too much fiddling in the bios.

OT: looking into some fancy setup for diskless boot over the network (i.e. completely rip-off from microvm and then add a pxe server here, an nginx there 🤣). When that is done I will finally have a nvidia (1050 2GB) machine running and will do a sanity check. Shotgun debugging FTW

[yangm97]

https://github.com/astro/microvm.nix/blob/0a3d48e06b8c04beb3de0a3283bd1ef29fe4a47d/nixos-modules/microvm/system.nix#L74

Maybe blacklisting drm here could be causing the chaos but I haven't tried overriding that yet.

[bnavetta]

I'm using virtio-vga-gl instead of GPU passthrough, but can confirm that re-enabling the drm module worked for me. This is what I've got in my VM's config:

boot.blacklistedKernelModules = lib.mkForce [ "rfkill" "intel_pstate" ];
boot.kernelModules = ["drm" "virtio_gpu"];

microvm = {
  hypervisor = "qemu";
  qemu.extraArgs = [
    "-device" "virtio-vga-gl"
    "-display" "sdl,gl=on"
    "-device" "qemu-xhci"
    "-device" "usb-mouse"
    "-device" "usb-kbd"
  ];
}

[astro]

@bnavetta Thank you for the snippet.

I have added the settings for qemu in c98c6202a6cca4fccd7437a366c82a06b9777d9e and bf6026f.