nixGL

nixGL copied to clipboard

Is there no way to make this a proper nixpkg? It feels clunky to install manually this way. I want this to be part of my configuration.

Well, I can push the derivation in nixpkgs, but you'll still have to call it manually to setup the desired driver.

I'll see if I can find something smart today, thank you for opening this issue, it will motivate me ;)

Actually it may be interesting if I provide a way to easilly wrap another program. So for example, a custom .config/nixpkgs/nix.conf would contain something such as:

   myNixGLNvidia = pkgs.nixGL { nvidiaVersion = ...; ... };
   myNixGLMesa = pkgs.nixGL { mesa = ...; };
   ....

   # I want blender to always use the intel driver
   # This will wrap `blender` executable with `nixGLMesa`
   blender = myNixGLMesa.wrap {pkg = super.blender};

   # Sometimes I'm doing really heavy 3D work with blender, let's wrap it with nvidiaBumblebee driver
   # Hence I'll have a `blender-fast` executable
   blenderNvidia = myNixGLNvidia.wrap {pkg = super.blender; suffix = "-fast";};
};

What are you thoughts about that?

my use case is selenium in a nix-shell, which does some funky stuff, however, other people don't have this problem on non-NixOS systems, so I cannot wrap it unconditionally

Sorry for the delay.

As a workaround, you can always do:

nixGL = fetchTarball "https://github.com/guibou/nixGL/archive/master.tar.gz"

myNixGL = (import "${nixGL}/default.nix") {
    nvidiaVersion = "...";
    nvidiaHash = "...";
    pkgs = pkgs;
}).nixGLNvidia;

Which will brings myNixGL package in scope. Depending on if you are in a nix-shell or in a nixos configuration file, you can put this package in the needed buildInputs or environment package list.

Actually it may be interesting if I provide a way to easilly wrap another program.

Not the OP, but to me this would sound pretty much ideal!

how's the process on this? would be nice to have a clean way to use programs that are affected by the gl issues configured in home-manager on non-nix ^^ Having an easy way to wrap things like alacritty would be nice

I'm also interested in this. Including nixGL as is in nixpkgs would already be good first step. Wrappers would be a nice second step in my opinion.

IIUC, modern nixpkgs is a flake, so all the inputs have to be listed explicitly and their hashes committed in a flake.lock file.

This means 10 gazillion Nvidia installer versions would have to be listed explicitly (the webpage http://download.nvidia.com/XFree86/Linux-x86_64/ does have them in a simple format, so merely copy-pasting the entire page should give a list).

This also means that the hack using /proc to detect the Nvidia version won't work anymore: a flake cannot have impure inputs, so one cannot use builtins.currentTime to force a re-read of that file.

One solution I'd suggest:

• hardcode the list of Nvidia driver versions
• place each driver in an attribute nixGL.nixGLNvidia.___VERSION___ so that they statically exist as outputs
• Do the /proc/ look-up and pick the right package at run-time, e.g. nixGLNvidiaAuto would be a shell script running version=$( /proc stuff ); nix-shell -p "nixGL.nixGLNvidia.$version" --run "$(printf "%q " "nixGLNvidia$version" "$@")"
• if the user knows their NVidia version ahead of time, they can directly use nixGL.nixGLNvidia.VERSION in their config, and possibly symlink that to nixGLNvidia or use a nixGL.nixGLNvidia.___VERSION___.wrap some-other-package.

@jsmaniac Thank you for your feedback.

hardcode the list of Nvidia driver versions

Yes, definitly yes. This will improves bulid time, reproducibility and everything. The issue is that we, nixGL authors, will have to be really reactive to new nvidia release and this will forces user to upgrade their nixpkgs often. So a fallback mecanism would have to be implemented.

Do the /proc/ look-up and pick the right package at run-time, e.g. nixGLNvidiaAuto would be a shell script running version=$( /proc stuff ); nix-shell -p "nixGL.nixGLNvidia.$version" --run "$(printf "%q " "nixGLNvidia$version" "$@")"

Yes! I do really dream about a single nixGL wrapper which does the install at runtime. However that's not trivial because you must ensure that the driver you are installing (at runtime) does match the libraries used in the installed binary.

a fallback mecanism would have to be implemented.

That's true. I guess keeping a nix expression similar to the current one, and having a list of known versions (with their flake.locked hashes (though I'm not sure if a flake's inputs can be described dynamically using a list)) would make it possible to still use a custom version? nix-shell has --arg that I've seen used, might allow to specify the version if it wans't known.

you must ensure that the driver you are installing (at runtime) does match the libraries used in the installed binary.

I didn't see where this happens in the code, could you give me a pointer to the relevant code or some comment? I think I slightly misunderstood something in that sentence but can't figure exactly what :smiley:

I didn't see where this happens in the code, could you give me a pointer to the relevant code or some comment? I think I slightly misunderstood something in that sentence but can't figure exactly what

This is not happening in the code right now, you are right.

The point is that you want the installer driver to match the running code. Here I see two uses cases:

• You have a "locally" defined "nix-shell" which includes nixGL as well as some other programs / libraries. In this context, nixGL is built with the same libraries as the other available programs, so right now it is fine.
• You have a system wide install of nixGL, so we can suppose that your system binaries does uses the same libraries.

This is the current situation and so far it works "enough".

With a "runtime" detection and install, we'll have:

• For the "local" nix-shell, the "runtime" nixGL must point to the "local" nixpkgs clone, to ensure the "runtime" installed drivers will match the "local" libraries.
• For the "system" installed nixGL, same thing, you must ensure that your current nix-channel does match the one of your system.

Oh, I see. If I'm understanding correctly, if my nixpkgs is an overlay mixing packages from 20.03 which I want to keep stable and some packages from 20.09 to get the latest and greatest, the nixGL should use 20.03's nixpkgs for the former and 20.09's nixpkgs for the latter?

If that's indeed the case, I don't see how it could be solved other than using a wrap which takes a package and a reference to its initial nixpkgs (to fetch the libraries there). That would be unfortunate, as wrap isn't the nicest UI for people managing their packages with nix-env -i (fine for me as I use a flake, but others might have issues).

@jsmaniac Yes, you got it right. The use case I'm having everyday at work is that I'm working on NixOS but I have different nixpkgs pinned for my clients. So each nix-shell for each client contains its own nixGL which depends on the "correct" version of the driver.

I tried turing nixGL into a flake (without having auto-detection etc.), and it works : https://github.com/jsmaniac/nixGL/blob/b93246bced824e54e3c2bc059af5c39379813904/flake.nix

However, I have to hard-code the choice between nixGLIntel and nixGLNvidia, and I have to hard-code a specific NVidia version. I guess making different outputs for the flake would allow multiple versions to be available in nix flake show (though I'm afraid doing so would mean that the driver for every Nvidia version would be downloaded just to evaluate the flake?).

I'm not sure how a flake would mix with providing a wrap function to be available in other flakes (e.g. someone's config, written as a flake).

@guibou Should the flake have a dummy derivation for defaultPackage.x86_64-linux, and provide another non-derivation output which represents the wrap function? Should this just be a regular .nix file which is somehow imported by another flake (e.g. someone's config, written as a flake)?

@jsmaniac

non-derivation output which represents the wrap function

The lib output of a flake woud do? I've seen mach-nix has recently added one: https://github.com/DavHau/mach-nix/blob/8a719adddf9ef6a043afe31f14dbbfcb147a7c25/flake.nix#L36

builtins.currentTime

This also prevents using nixGLNvidia in another flake with nixGL as a non-flake input: https://github.com/newkozlukov/dotfiles/blob/1e0bc11906602104eaeb71fcd3c6704d8b24c2b2/flake.nix#L8 (the reason to use flakes here is to pin revisions using flake.lock, removing the need for e.g. niv)

driver for every Nvidia version would be downloaded just to evaluate the flake

Not sure what is meant by "evaluating" a flake, but AFAIU it is one of the main ideas behind flakes that nix flake show&c be provided with enough metadata about inputs/outputs that they do not require (and explicitly prohibit) any computation, do not cause derivations to be built, nor do they even require checking out the full source

@guibou

... and this will forces user to upgrade their nixpkgs often

Turning nixGL into a flake would actually allow to keep it outside nixpkgs - thus having its own release cycle - and still make it totally re-usable by other users without much effort (e.g. interactively nix run github:guibou/nixGL#nixGLIntel, or adding to the inputs of their own flakes)! Further on "updating nixpkgs", flake users can set inputs.nixGL.inputs.nixpkgs.follows = "nixpkgs" to get nixGL built with the same nixpkgs revision as the rest of their environment (if I am not mistaken, that is)

I haven't looked into auto-detection, but I did spend some time to make this easy to use for wrapping applications, for now I created the following overlay:

final: prev:

let
  lib = prev.lib;

  nixGL = import inputs.nixGL { pkgs = final; };
  wrapWithNixGL = wrapper: package:
    let
      getBinFiles = pkg:
        lib.pipe "${lib.getBin pkg}/bin" [
          readDir
          attrNames
          (filter (n: match "^\\..*" n == null))
        ];

      wrapperBin = lib.pipe wrapper [
        getBinFiles
        (filter (n: n == (lib.getName wrapper)))
        head
        (x: "${wrapper}/bin/${x}")
      ];

      binFiles = getBinFiles package;
      wrapBin = name:
        final.writeShellScriptBin name ''
          exec ${wrapperBin} ${package}/bin/${name} "$@"
        '';
    in final.symlinkJoin {
      name = "${package.name}-nixgl";
      paths = (map wrapBin binFiles) ++ [ package ];
    };

  wrappers = let replacePrefix = replaceStrings [ "wrapWithNixGL" ] [ "nixGL" ];
  in lib.genAttrs [
    "wrapWithNixGLNvidia"
    "wrapWithNixGLIntel"
    "wrapWithNixGLDefault"
  ] (name: wrapWithNixGL final.${replacePrefix name});
in {
  inherit (nixGL) nixGLNvidia nixGLIntel nixGLDefault;
  inherit wrapWithNixGL;
} // wrappers

This first overlay is shared between several different systems, then I use it in another overlay like this (defined per system):

final: prev:

prev.lib.genAttrs [
  "kitty"
  "qutebrowser"
] (name: prev.wrapWithNixGLIntel prev.${name})

I think if we integrate something like this + potentially auto-detection and make it a flake that would make it easy to consume.

For us, we have multiple users (say, 10 developers) and each needs to be able to run some produced binary and the setups aren't uniform at all: some people on nvidia, some on intel and versions vary.

I was unable to use the above code as-is because of builtins.currentTime usage which is not allowed in the flake for example and we'd have to pick a version up front anyway.

Instead, I did something like this:

        nixGL-overlay = self: super: {
          wrapWithNixGLCommand = { wrapper, package, bin_name ? package.name }:
            self.runCommand bin_name {

              buildInputs = [ self.makeWrapper ];
            } ''
              mkdir -p "$out"/bin
              cat << EOF > "$out"/bin/${bin_name}
              #!${self.bash}/bin/bash
              set -e
              nixGL="\$(NIX_PATH=nixpkgs=${inputs.nixpkgs} nix-build ${inputs.nixGL} -A ${wrapper} --no-out-link)"/bin/*
              if [ -z \''${nixGL+x} ]; then
                >&2 echo 'Failed to install nix OpenGL wrapper, trying to run ${bin_name} anyway...'
                exec -a ${bin_name} ${
                  self.lib.getBin package
                }/bin/${bin_name} "\$@"
              fi
              exec -a ${bin_name} \$nixGL ${
                self.lib.getBin package
              }/bin/${bin_name} "\$@"
              EOF
              chmod +x "$out"/bin/${bin_name}
            '';

          wrapWithNixGL = package:
            let
              wrappers = {
                "nvidia" = "nixGLNvidia";
                "nvidia-bumblebee" = "nixGLNvidiaBumblebee";
                "nvidia-vulkan" = "nixVulkanNvidia";
                "intel" = "nixGLIntel";
                "intel-vulkan" = "nixVulkanIntel";
              };
            in self.lib.mapAttrs' (n: v:
              self.lib.nameValuePair "${package.name}-${n}"
              (self.wrapWithNixGLCommand {
                inherit package;
                wrapper = v;
              })) wrappers;
        };

This generates attributes for the package for each of the drivers. If there was now a way to detect which command is likely the necessary one, this would be much better and we could just detect in the wrapper.

We probably won't be actually sticking to this approach however. It doesn't work in the nix shell where user drives the build themselves and gets the unwrapped binary. It also adds non-insignificant overhead to the command start.

I think what I'll end up hacking in the end is checking if any of the nixGL* commands are installed and invoking one. Then I put the burden on the user to install the right one. In shell, they will have to run nixGL… <binary> explicitly which is kind of annoying: maybe there should be nixGL* command detection in the binary itself and it then spawns the actual process with it. That may be better and removes the need for separate wrapper in the expression.

Thank you for this discussion. Many points

• your approach with a wrapper which calls nix-build does have a limitation. I suppose that input.nixpkgs contains the path to your nixpkgs clone. If you introduces an overlay to nixpkgs which override an important dependency of nixGL, it will build a different version.
• As you observed, the call to nix-build is slow.

One solution there is instead of using nix-build with a nix attribute, you can use nix-build with a drv. Quick diff:

              nixGL="nix-build  ${wrapper.drvPath} --no-out-link)"/bin/*

This will ensure that the right derivation is built and nix-build from a derivation is way faster.

However, this won't work for the nvidia driver, because you need to know at build time the version of the nvidia driver you care about.

One solution, which is crazy but may work, is to generate the list of all possible nvidia drivers, and generates as many derivation as possible and do the auto detection at runtime in nixGL itself and call the right nvidia driver. However it means that nixGL would have to be reinstalled and perhaps patched if the user want an nvidia driver which is more recent that the one we know about.

About your shell issue, one thing which can be done is to refactor nixGL so that the list of environment variables which needs to be exported can be exported in the nix-shell shellHook. It will work for nvidia and intel, but not for "hybrid" config, where users will have to explicitly use bumblebee or alternate launcher.

I'll propose that we do the following refactoring:

• refactor all the wrapper so we can easily export the environment variables in a nix-shell, and keep "auto configuration" for nix-shell
• write a list of all "available" at time nvidia drivers + source + hash
• write a super nixgl program which contains path to all the nix derivation for all possible drivers and detect at runtime the one which is needed.

Note: we can even write a "fallback" for nvidia drivers which, if the driver version number is not known, we try to nix-build it using the original approach.

I'll give a try to this approach this afternoon. (ping me in thee month if I did not deliver... ;)

Thanks for the comments, I'll check them in detail tomorrow.

w.r.t. nvidia versions: I think this isn't really a problem honestly. It seems easy to automate updating a list of all the driver versions along with having a small program that lets someone do it themselves. It could even live in a separate repository as a flake. This of course only works if you are able to update nixGL when necessary which is not really a restriction for us though it does sound annoying.

At least in the short term, I think we will go with:

• have user install appropriate nixGL command once
• make a simple wrapper that finds installed nixGL comand and execs the actual underlying process with it
• have user reinstall nixGL if something breaks due to updated system driver

This works for us but I realise it's quite specific. Automatic detection of driver could at least eliminate the manual install part.

Basically whatever we go with, it will probably be invoking nixGL externally and not be part of the build process as jt inherently has to do some runtime detection.

make a simple wrapper that finds installed nixGL comand and execs the actual underlying process with it

I forgot to mention why I'd want to do this: having such detection/wrapper embedded into the binary produced by our build system (cargo in this case) allows the user to invoke the binary directly even if they produced it in manual build via cargo while in a nix shell: it removes the need for separate logic in shell and when built via nix.

Of course it requires minor work and quite specific hack to the binary so it's not a general solution.

At least in the short term, I think we will go with:

* have user install appropriate nixGL command once

* make a simple wrapper that finds installed nixGL comand and execs the actual underlying process with it

* have user reinstall nixGL if something breaks due to updated system driver

How about this as a solution: All of the wrappers have a means of detecting (via env vars or otherwise) if they're already in an environment that has been touched by another wrapper, and if so just do nothing. Ie. all of the wrappers would set an env var NIXGL_WAS_HERE_FLAG=true. If any wrapper detects that NIXGL_WAS_HERE_FLAG has been set, then it just becomes a pass-through.

That way you could do things like

# wrapped-alacritty = nixGL alacritty              as packaged upstream
$ nixGLIntel wrapped-alacritty                     # or nixGLNvidia, or nixGLNvidia.<driver version>, and so on...

and it would be the same, eg. nixGLIntel (nixGL alacritty) == nixGLIntel alacritty. This way applications could be packaged upstream with the nixGL default wrappers, but still be infinitely customizable by end users. And if a user would always like alacritty to be opened in a specific configuration, they could do so with overrideAttrs, a custom derivation, or simply a shell alias.

I'd like to throw out there that it's also important to find a solution that "just works" for users launching programs from desktop launchers. To me that indicates that ideally applications would be packaged with the default wrappers.

NIXGL_WAS_HERE_FLAG sounds OK; but I think we still would like no make it possible to have something just work without user having to explicitly invoke nixGLIntel or whatever. I think one needs auto-detection.

As a follow-up to my own comment, I've implemented a simple wrapper that just checks if nixGL* command is on the system and re-invokes itself with it, setting some flag like --skip-nix-gl-check (akin to NIXGL_WAS_HERE_FLAG). This only works for this one case though and not when user has multiple graphics setups and such but it was good enough for us. Maybe it helps someone else too. The program in question is called isim fwiw.

use std::ffi::CString;
use std::os::unix::ffi::OsStrExt;

const KNOWN_NIX_GL: [(&str, &str); 5] = [
    ("nvidia", "nixGLNvidia"),
    ("nvidia-bumblebee", "nixGLNvidiaBumblebee"),
    ("nvidia-vulkan", "nixVulkanNvidia"),
    ("intel", "nixGLIntel"),
    ("intel-vulkan", "nixVulkanIntel"),
];

fn instructions() -> String {
    let mut s = String::new();
    s.push_str("It looks like no nixGL driver is installed on your system.\n");
    s.push_str("You need this to run graphical applications built with help of nix.\n");
    s.push_str("At the top level of the engine repository, run one of these commands, depending on your driver.\n");
    s.push('\n');
    for (driver, _) in KNOWN_NIX_GL.iter() {
        s.push_str(&format!("{} driver:\n", driver));
        s.push_str(&format!(
            "\t$(nix-build default.nix -A install-nix-gl-{} --no-out-link)/bin/install-nix-gl\n",
            driver,
        ));
    }
    s.push('\n');
    s.push_str("If you don't know which driver you need or have any other issues, ask on Slack.\n");
    s.push_str("If you're a super-user and know what you're doing, you can use --skip-nix-gl-check flag.\n");
    s
}

/// This program tries to detect one of known binary names on the system. If
/// found, it re-invokes itself with an additional flag to skip this check in
/// the future. If not found, it prints a warning with explanation.
///
/// The warning could be made optional by detecting (somehow) whether this is a
/// nix-built isim. However, I think it's OK to have it anyway: nix-supported
/// (nix-shell or nix-build) should be the only way Soon™ so if one sees this,
/// they are probably doing something strange like running a version compiled
/// with some manual set-up producing different results.
pub fn try_ensure_nix_gl() {
    let nix_gl_binary = KNOWN_NIX_GL
        .iter()
        .find_map(|(_, bin_name)| which::which(bin_name).ok());
    match nix_gl_binary {
        Some(nix_gl_binary) => {
            let nix_gl_binary = CString::new(nix_gl_binary.clone().into_os_string().as_bytes())
                .unwrap_or_else(|e| {
                    panic!(
                        "Could not get CString out of {} due to {}",
                        nix_gl_binary.display(),
                        e
                    )
                });

            let current_isim_path =
                std::env::current_exe().expect("Couldn't get the current isim location.");
            let current_isim_name = CString::new(
                current_isim_path
                    .file_name()
                    .expect("Couldn't get isim binary name.")
                    .as_bytes(),
            );

            let current_isim_path = CString::new(current_isim_path.as_os_str().as_bytes());
            let mut args = vec![
                // Set the name of the process to whatever the isim binary is
                // called.
                current_isim_name,
                // Invoke the real location of isim.
                current_isim_path,
                CString::new("--skip-nix-gl-check"),
            ];
            // Rest of arguments we pass as-is, remembering to skip argv[0]
            // which just contains the isim invocation line: we already pass the
            // full path above so we only care about anything that was passed in
            // afterwards.
            args.extend(std::env::args().skip(1).map(CString::new));
            let args = args.into_iter().map(|x| x.unwrap()).collect::<Vec<_>>();

            // TODO: How to set the process name like `exec -a`?
            nix::unistd::execv(&nix_gl_binary, &args).unwrap();
        }
        None => {
            eprintln!("{}", instructions());
            eprintln!("Continuing to run isim as-is anyway...")
        }
    }
}

Thank you for you comments.

Unfortunately, I do have too much things ongoing right now and I do not affect enough time to nixGL as I would like.

I'll read and easily approve any PR you may submit regarding this issue.

Hopefully I'll have more time in the future to work on this.

The approach (ugly hack) I had to getting this to work under flakes is like so:

homeConfigurations = {
  exampleHost = inputs.home-manager-unstable.lib.homeManagerConfiguration {
    configuration = { config, pkgs, lib, ... }: {
      home.packages = let
        nixGLNvidiaScript = pkgs.writeShellScriptBin "nixGLNvidia" ''
          $(NIX_PATH=nixpkgs=${inputs.nixpkgs-unstable} nix-build ${inputs.nixgl} -A auto.nixGLNvidia --no-out-link)/bin/* "$@"
        '';
        nixGLIntelScript = pkgs.writeShellScriptBin "nixGLIntel" ''
          $(NIX_PATH=nixpkgs=${inputs.nixpkgs-unstable} nix-build ${inputs.nixgl} -A nixGLIntel --no-out-link)/bin/* "$@"
        '';
        nixVulkanIntelScript =
          pkgs.writeShellScriptBin "nixVulkanIntel" ''
            $(NIX_PATH=nixpkgs=${inputs.nixpkgs-unstable} nix-build ${inputs.nixgl} -A nixVulkanIntel --no-out-link)/bin/* "$@"
          '';
        nixVulkanNvidiaScript =
          pkgs.writeShellScriptBin "nixVulkanNvidia" ''
            $(NIX_PATH=nixpkgs=${inputs.nixpkgs-unstable} nix-build ${inputs.nixgl} -A auto.nixVulkanNvidia --no-out-link)/bin/* "$@"
          '';
      in with pkgs; [
        nixGLNvidiaScript
        nixGLIntelScript
        nixVulkanIntelScript
        nixVulkanNvidiaScript
      ];
    };
    system = "x86_64-linux";
    homeDirectory = "/home/user";
    username = "user";
  };
};

where the inputs are my flake inputs for nixpkgs-unstable, home-manager, and this repository. For some context: I'm running on Fedora with a single user nix install. I use flakes and home-manager. My laptop is dual-gpu, so normally i would run opengl apps with nixGLIntel, but for the occasions when I want to offload to the nvidia GPU, I'd have to set the according environment variables, then run the nixGLNvidia script.

Because I use flakes, if the channels that get updated by nix-channel get too out of sync with the commit i build my home-manager generations on, nixGL stops working (i tested this by building my home-manager config with 21.05 and leaving my nix-channel channels on unstable).

This is an ugly hack, and might not be too relevant to this particular issue, but I figure it's worth putting here just in case someone else like me found this issue searching for something like this.

~~Re flake: what about just providing an impure flake and instructing users to install with nix profile install --impure? I encountered this impure flag running-unfree-packages.~~ Edit: I read this thread in detail and realized working around with --impure is OT

@edrex I'll test an approve any PR related to this issue.