Integrating Fex support

[rowanG077]

With fex + vulkan seeming to be pretty close on fedora asahi I think we should start thinking about how we can support fex on nixos. As far as I see it a few different things need to happen for that:

• Have a working FEX build
• Be able to run a 4k page microVM, probably using the same method as fedora does it.
• Run the rootFS in the VM

I personally have little experience how rootfs + krun functions so this is more to get the discussion started. I'm willing to work on this so if anyone has any pointers to achieve this that would help as well.

[rowanG077]

Pinging @RossComputerGuy as you have packaged krun aka muvm upstream. Currently it crashes for me but maybe that is known.

[RossComputerGuy]

This is definitely a good idea and I can look into it when I have a moment. (I should rename krun to muvm in nixpkgs, didn't know about the rename until recently)

[rowanG077]

This is a fex derivation, I can't test it because you need a 4K pagesize system to run it. For which we will need muvm. If you disable tests though it fully builds and you can at the very least run FEXConfig.

{ lib, stdenv, clangStdenv, fetchgit, cmake, clang, ninja, pkg-config, lld, llvm
, llvmPackages, openssl, glibc, autoPatchelfHook, nasm, qt5, python, setuptools
, libclang
}:

clangStdenv.mkDerivation rec {
  pname = "fex-emu";
  version = "2410";

  src = fetchgit {
    url = "https://github.com/FEX-Emu/FEX.git";
    rev = "FEX-${version}";
    sha256 = "sha256-btzb7OGa3M89wDn8AK/iocT3GLY1mB3cZ0iuWAyNIFc=";
    fetchSubmodules = true;
  };

  nativeBuildInputs = [
    cmake
    clang
    ninja
    pkg-config
    nasm
    qt5.wrapQtAppsHook
    # required with "-DUSE_LINKER=lld"
    # lld
    # autoPatchelfHook
  ];

  buildInputs = [
    llvm
    python
    setuptools
    libclang
    openssl.dev
    qt5.full
  ];

  cmakeFlags = [
    # upstream recommends this but binaries have linker issues
    # libstdc++.so.6: cannot open shared object file: No such file or directory
    # autoPatchelfHook fixes that for the output binaries but test are run
    # by cmake prior to that
    # "-DUSE_LINKER=lld"
    "-DENABLE_ASSERTIONS=FALSE"
    "-DBUILD_TESTS=FALSE"
    "-DOVERRIDE_VERSION=FEX-${version}"
  ];

  meta = with lib; {
    description = "FEX - Fast x86 emulation frontend";
    homepage = "https://github.com/FEX-Emu/FEX";
    license = licenses.mit;
    platforms = platforms.linux;
    maintainers = with maintainers; [ rowanG077 ];
  };
}

[Lemm1]

This is a fex derivation, I can't test it because you need a 4K pagesize system to run it. For which we will need muvm. If you disable tests though it fully builds and you can at the very least run FEXConfig.

The problem with FEX is to create RootFS, there are prebuilds for fedora, arch and ubuntu. So for nixos you will need to create your own. I've tried to do it few months ago, but it wasn't that easy as creating FEX package. Also in order to make something distributable it will require some nix wizard to somehow either automate RootFS creation on package build or to have prebuilt RootFS distributed like FEX do it for some distros. I personally gave up on that and started using distrobox for FEX

[rencire]

I was doing some basic research and found this post that could be helpful for those who want to pursue this: https://old.reddit.com/r/AsahiLinux/comments/1hd6wkg/fexemu_on_asahi_debian/m1u7uxr/

[mkurz]

Hi,

we made fex + muvm + steam work on arch linux arm, now hosted under https://asahi-alarm.org/ (GitHub: https://github.com/asahi-alarm/asahi-alarm/ and the other repos in that org).

To build the rootfs we forked https://github.com/FEX-Emu/RootFS - see how we set up the config in this branch: https://github.com/asahi-alarm/RootFS/tree/asahi-alarm (or look at the diff) To build the erofs image we now can just run: ./build_image.py -m 2G -no-repack-tar -no-repack-squashfs Configs/Arch.json cache_dir rootfs_dir (you might also want to disable kvm if needed: -disable-kvm) on an Ubuntu 24.04 x86_64 machine where we have apt-file python3-git python3-requests qemu-utils pigz guestfs-tools cloud-image-utils erofs-utils linux-image-kvm installed.

And here is how we build the overlays: https://github.com/asahi-alarm/mesa-overlay/blob/main/Dockerfile (running with docker build --no-cache=true --output=build .) - using the mesa PKGBUILD - which is 1:1 what Fedora is using (by the means of whichs make flags they are using and which files the install)

Here are the PKGBUILDs for all our packages: https://github.com/asahi-alarm/PKGBUILDs

Maybe this helps, have a nice day!

[nrabulinski]

I'm currently running NixOS with Asahi kernel on my M1 MacBook Pro and got muvm with x86 emulation and gpu acceleration working. It's not super clean and sadly x86 emulation doesn't work in the Nix sandbox yet which I consider a must, but I'll be submitting the required patches to muvm first before cleanly packaging the rest

[rowanG077]

@nrabulinski Awesome, yeah I saw your issues and discussion on muvm! I look forward to testing your work.

[nrabulinski]

In the meantime it'd be awesome if someone could get a cross-compile working to build x86_64-linux version of mesa-asahi-edge on aarch64-linux. I worked around it for my usecase by simply using a native x86_64-linux builder, but that's not sustainable and not everyone will have that luxury. An alternative would be to require people to first "bootstrap" by building muvm and fex first, registering them as x86 interpreter, and then building the graphics driver as an x86_64 system, but that's not very elegant. Is there a nixos-on-asahi community somewhere already? If not, maybe people would be interested in creating a dedicated matrix room at nixos.org?

[nrabulinski]

Update: my PRs to muvm are open and waiting for review. Also, with the help of @yuyuyureka I managed to cross-compile mesa for x86 on my asahi machine. I haven't yet tried packaging fex but we're closer than ever to proper x86 app support on NixOS asahi :) The steam wrapper will still have to wait, as well as running x86 programs inside the Nix sandbox, but for day-to-day use it should make nixos with asahi kernel much more usable

[nrabulinski]

I've created https://github.com/nrabulinski/nixos-muvm-fex for my muvm+fex experiments, before starting to upstream those efforts. If someone just wants to run x86 linux apps on their asahi machine, well that's sufficient I'm doing so right now. GUI apps work just fine, with hardware acceleration :) The current limitations are: you can't use muvm+fex as binfmt handler, even if you could, muvm doesn't like the nix sandbox. You also cannot use nix from inside the muvm guest, that's also work that's in progress as we speak

We also have official matrix channel, feel free to join at #asahi:nixos.org

[rowanG077]

You are a true legend @nrabulinski!

[Yeshey]

I've created https://github.com/nrabulinski/nixos-muvm-fex for my muvm+fex experiments, before starting to upstream those efforts. If someone just wants to run x86 linux apps on their asahi machine, well that's sufficient I'm doing so right now. GUI apps work just fine, with hardware acceleration :) The current limitations are: you can't use muvm+fex as binfmt handler, even if you could, muvm doesn't like the nix sandbox. You also cannot use nix from inside the muvm guest, that's also work that's in progress as we speak

We also have official matrix channel, feel free to join at #asahi:nixos.org

Can this be used to run x86 software like steamcmd in a KVM server like Oracle's Ampare A1 arm servers?

[RossComputerGuy]

I think so, though it shouldn't be exactly required if you're running with a 4k page size. I've been trying with regular steam on Ampere Altra Max M128-26.

[Yeshey]

I think so, though it shouldn't be exactly required if you're running with a 4k page size. I've been trying with regular steam on Ampere Altra Max M128-26.

cool, yeah, Oracle Ampare A1 does have 4K page size, that means muvm is not needed then, right? And how are you running steam, or x86 packages? Are you getting them from nixpkgs? I have a project for box64 where I added an overlay to allow installing x86 apps on aarch64 with:

  imports = [ inputs.box64-binfmt.nixosModules.default ];
  nixpkgs.overlays = [ inputs.box64-binfmt.overlays.default ];

  environment.systemPackages = [
    pkgs.x86.steamcmd
    pkgs.x86.wineWowPackages.stable
    pkgs.x86.katawa-shoujo
  ];

But that is kinda hacky, and sets up a bunch of things for qemu emulation to allow building:

boot.binfmt.emulatedSystems = ["i686-linux" "x86_64-linux" "i386-linux" "i486-linux" "i586-linux" "i686-linux"];
nix.settings.extra-platforms = ["i686-linux" "x86_64-linux" "i386-linux" "i486-linux" "i586-linux" "i686-linux"];

And then when removing it from my config it might happen that it ends up with wrong architecture binaries in the middle of the process and it trips out, couldn't build configuration, and I had to redeploy the server, I don't have time to try my own flake again rn ahahah

idk if I was overengineering ahaha

[RossComputerGuy]

I have not gotten Steam working due to muvm failing to pass through the NVIDIA GPU I have. I've also tried getting Steam in but libX11.so.6 fails to load for some reason.

[rowanG077]

Now that muvm is upstream I thought I'd try it. I get a panic but it still launches the command.

> nix-shell -p muvm
[nix-shell:~]$ RUST_BACKTRACE=1 muvm sh

thread 'gpu worker' panicked at src/devices/src/virtio/gpu/virtio_gpu.rs:254:14:
Rutabaga initialization failed!: ComponentError(-19)
stack backtrace:
   0: rust_begin_unwind
   1: core::panicking::panic_fmt
   2: core::result::unwrap_failed
   3: devices::virtio::gpu::worker::Worker::work
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Using default interface naming scheme 'v257'.
sh-5.2$ /nix/store/vlzzppxvd7swdy67kj3avr5r9147llzf-udev-rules/51-android.rules:1101 Unknown group 'adbusers', ignoring.

sh-5.2$ 

Is there anything I need to do to get this working? I'm also interested in running steam.

[RossComputerGuy]

That's the same issue I see on my Ampere desktop with an NVIDIA RTX 5070. I don't see that issue on my M1 Pro MBP.

[rowanG077]

I'm using an M2 Max for reference. Using this repo + nixos unstable. I use the experimentalGPUInstallMode set to replace if it matters.

[Liamolucko]

Now that muvm is upstream I thought I'd try it. I get a panic but it still launches the command.

nix-shell -p muvm [nix-shell:~]$ RUST_BACKTRACE=1 muvm sh

thread 'gpu worker' panicked at src/devices/src/virtio/gpu/virtio_gpu.rs:254:14: Rutabaga initialization failed!: ComponentError(-19) stack backtrace: 0: rust_begin_unwind 1: core::panicking::panic_fmt 2: core::result::unwrap_failed 3: devices::virtio::gpu::worker::Worker::work note: Some details are omitted, run with RUST_BACKTRACE=full for a verbose backtrace. Using default interface naming scheme 'v257'. sh-5.2$ /nix/store/vlzzppxvd7swdy67kj3avr5r9147llzf-udev-rules/51-android.rules:1101 Unknown group 'adbusers', ignoring.

sh-5.2$

Is there anything I need to do to get this working? I'm also interested in running steam.

That looks like the error I get when I'm not using Asahi's patched version of virglrenderer:

virglrenderer = prev.virglrenderer.overrideAttrs (old: {
  src = final.fetchurl {
    url = "https://gitlab.freedesktop.org/asahi/virglrenderer/-/archive/asahi-20250424/virglrenderer-asahi-20250424.tar.bz2";
    hash = "sha256-9qFOsSv8o6h9nJXtMKksEaFlDP1of/LXsg3LCRL79JM=";
  };
  mesonFlags = old.mesonFlags ++ [ (final.lib.mesonOption "drm-renderers" "asahi-experimental") ];
})

That's the same issue I see on my Ampere desktop with an NVIDIA RTX 5070. I don't see that issue on my M1 Pro MBP.

I'm not sure why it's working on your M1 Pro (maybe you are using the patched version and forgot about it?), but I think it's expected for it not to be working with an NVIDIA GPU: it seems like muvm only supports GPU acceleration using DRM native context, which isn't supported on NVIDIA GPUs (https://github.com/AsahiLinux/muvm/blob/main/README.md?plain=1#L67).

[rowanG077]

@Liamolucko Can confirm that with that patched virglrenderer I no longer see the panic, thanks!. But honestly at this point I'm kind stumped. How can I get an x86 nixos environment here, or perhaps even a fedora environment? Or is that not yet possible?

[psanford]

I was able to get steam running along with a few games I have handy (Void War, Enter the Gungeon).

I had to hack things up pretty bad so I don't have a nice set of instructions yet. I will try to cleanup my setup this week and then share out instructions.

In the meantime here's briefly what I did:

• running muvm with the overlays provided by fedora is sufficient to get Fex working with basic (non-gui) x86 binaries
• The Steam wrapper scripts work in this environment but then steam segfaults and it wasn't clear why. I decided it would be easier to setup a fedora rootfs where I understood better how things should work
• I launched a fedora docker image, installed the asahi yum.repo.d repos, and then installed the asahi specific packages (mesa)
• I ran podman export ... to make a rootfs directory
• I patched muvm to mount this instead of /.
• I also patched muvm so that it would work even without / mounted (it makes a lot of assumptions that / is the host's /)
• I passed through MESA_LOADER_DRIVER_OVERRIDE=asahi
• This was enough to get glxgears working
• I extracted steam from the .deb and ran its install script
• There was as issue with the wrapper script getting hung running cat. I assume this is something very silly but I just killed cat to get it to continue
• Steam worked

[Zynh0722]

@psanford I'd absolutely love to put in some leg work in getting something that functions for nix, even if it's not something that fits directly in this repository. Unfortunately I'm not quite familiar enough with Fedora environments to reproduce your steps, and I couldn't find a way to contact you directly haha.

If you'd be willing to help me get through some of the gaps in my knowledge that would be incredible.

I also couldn't find anything else within this thread that got me anywhere useful, but if there is already something that exists that is largely plug and play for nixos users that people know about that would be great too haha

edit: at this point I just want something that works even a little bit to play games on the go haha

[psanford]

Sorry it took a bit for me to get back to this. I cleaned up my muvm patches so they should be usable for others now. I have dropped basic instructions for running this here: https://github.com/psanford/muvm/blob/alt-rootfs/NIXOS-APPLE-SILICON.md

[neobrain]

Thanks for writing up instructions! Just curious since I stumbled upon this the other day on Fedora, does your RootFS bundle all locales? It turns out not bundling any of the locale indicated by any LC_ env variable will add a 5 second delay to pressure-vessel, which adds up to a very noticeable slow down in start up Steam itself and games launched from Steam.

If locales aren't provided, pv-adverb[449]: W: Container startup will be faster if missing locales are created at OS level will be printed in webhelper-linux.txt and console-linux.txt.

[slendidev]

@psanford Hello, thanks for the guide. I got it to kinda work, had to add this flag: --mount run:/run/user:/run/muvm-host/run/user. glxgears opens, renders a frame, but then nothing else. Know anything about this? Thank you. Steam also opens up a black window, which I think is supposed to be the login window.

[psanford]

Do you have the same mesa version inside of the VM as you do outside?

[slendidev]

Do you have the same mesa version inside of the VM as you do outside?

Nope. On my host I have:

OpenGL version string: 4.6 (Compatibility Profile) Mesa 25.2.3

And in the fedora rootfs:

OpenGL version string: 4.6 (Compatibility Profile) Mesa 25.2.1

Know a way to get them to match?

[rowanG077]

I followed the tutorial and glxinfo and glxgears just hang. I can run basic commands like ls no problem. But there does seem to be a problem as I cannot run either in my normal terminal. I didn't notice this before.

edit: Update to most recent unstable, glxinfo works in native shell, but I can't enter the nix-shell inside the muvm repo it fails with:

<jemalloc>: Unsupported system page size
<jemalloc>: Unsupported system page size
<jemalloc>: Unsupported system page size

[waltmck]

A rootfs makes sense for FHS distros where programs look for specific locations in /lib to link against, but why is it necessary on NixOS? Shouldn't it be possible to pass the required x86 libraries as buildInputs (using patchelf where necessary) like we do normally, and have the two architectures' libraries coexist in the Nix store? It should even be possible to make a nix-ld environment to run unpatched x86 executables.

What is this approach missing?