Support Nix depexts with `opam env`

[RyanGibb]

Nix doesn't install packages in the traditional sense -- instead it puts them in a store and makes them available through environment variables. I.e., nix-shell -p gcc will drop us into a shell with gcc in the $PATH.

We can set appropriate environment variables with Opam to make system dependencies (depexts) available with Nix. Similar to how nix-shell works under the hood, we create a Nix derivation such as

{ pkgs ? import <nixpkgs> {} }:
with pkgs;
let
  packages = [ gmp ];
  inputs = with buildPackages; packages ++ [ pkg-config ];
in
stdenv.mkDerivation {
  name = "opam-nix-env";
  nativeBuildInputs = inputs;

  phases = [ "buildPhase" ];

  buildPhase = ''
vars=("NIX_CC" "NIX_CC_FLAGS" "NIX_CFLAGS_COMPILE" "NIX_CC_WRAPPER_TARGET_HOST_x86_64_unknown_linux_gnu" "NIX_LDFLAGS" "PKG_CONFIG_PATH")
for var in "''${vars[@]}"; do
  escaped="$(echo "''${!var}" | sed -e 's/^$/@/' -e 's/ /\\ /g')"
  echo "$var    =       $escaped        Nix" >> $out
done
echo "PATH      =       $PATH   Nix"
  '';

  preferLocalBuild = true;
}

Which we can build to output a file with environment variables that make depexts available, in Opam's environment variable format. This file is a Nix store root, so it's dependencies won't be garbage collected by Nix until the file is removed.

This approach came from conversations with @dra27 and is distinct from previous approaches in that it supports providing development environment, which imperative installations with Nix don't https://github.com/ocaml/opam/pull/5332#issuecomment-1710409758; and doesn't require the user to manage the environment outside of Opam, which would lead to a different workflow https://github.com/ocaml/opam/pull/5942.

Initial experiments were done using a package to set such environment variables https://github.com/RyanGibb/nix.opam. However, in order to work with generic depexts (as opposed to just conf packages), to avoid cyclic dependencies (if conf packages depend on nix.opam but nix.opam depends on conf packages), and due to opam package sandboxing (nix derivation fetching would require opam sandboxing to be disabled or the nix store to be 'primed'), it's better implemented as a depext mechanism.

This has been tested and successfully creates an environment to provide packages conf-gmp and conf-libseccomp, as well as depexts directly.

[RyanGibb]

Rebased on master

[RyanGibb]

Looks like I'm running into https://github.com/NixOS/nix/issues/10587 with the Docker image

[RyanGibb]

Rebased on master -- and could do with another depext CI run if it needs re-approval!

[RyanGibb]

Okay! The CI is failing only due to https://github.com/ocaml/opam-repository/pull/26261 (I've got it working locally with my branch of opam-repository)

[RyanGibb]

Anyone wanting to play around with this on a NixOS system can try:

$ nix shell github:RyanGibb/nixos/#legacyPackages.x86_64-linux.nixpkgs.opam
$ opam --version
2.3.0

Or add it to your own overlay with:

{
  nixpkgs.overlays = [
    (final: prev: {
      opam = prev.opam.overrideAttrs (_: {
       src = final.fetchurl {
          url = "http://ryan.freumh.org/software/opam-full-2.3.0-nixos-depexts.tar.gz";
          sha256 = "sha256-mRxxZtWFgQ8v1szVq5g5+qVqa+OffoG1aHzGUiMMvT0=";
        };
        version = "2.3.0";
      });
    })
  ];
}

This URL is just the https://github.com/RyanGibb/opam/tree/nixos-depexts-2.3.0 branch with vendored dependencies since I couldn't get them working in Nix,

[RyanGibb]

David made an observation in today's Opam dev meeting that this PR will not remove depexts from the Nix derivation until the next install. This might effect reproducibility in some circumstances, but this isn't a huge deal as if Nix users want full reproducibility they can use opam2nix.

[RyanGibb]

Thanks for the comments and commits @kit-ty-kate ! I'm glad it was useful :-) Apologies I haven't had a lot of time to work on this PR recently

[kit-ty-kate]

@RyanGibb Do you know why autopin.test fails? Looking briefly at the changes i can't really find the cause

[RyanGibb]

Hmm, I'm not sure. It seems the depext mechanism isn't being triggered when the test thinks it should, but I can't figure out why. I think this test is using the Dummy depext mechanism which (I think?) doesn't list installed packages as installed so retriggers the depext mechanism every time. I don't know why these changes would effect that behaviour, since we don't touch the Dummy path. It's also strange that this is the only failing test.

[RyanGibb]

I'm wondering if it could be something to do with the handling of available/required/installed packages, and am thinking of changing packages_status to return OpamSysPkg.status.

[kit-ty-kate]

I found the issue. One of the functions used all_packages instead of new_packages. I gotta say that i find these (old_packages, new_packages, all_packages) naming scheme pretty confusing. What do they refer to? What's new, old and all?

[RyanGibb]

What do they refer to? What's new, old and all?

So to support Nix depexts we have to pass the not only the packages we want to install, but also the old_packages which are already in the derivation, since we recreate it from scratch each time. https://github.com/ocaml/opam/pull/5982/files#diff-6c0a6590c49c5a229f469d627b9e9484f1b20c1fb5fb1c8102d64a3d2c1d09b6R476

We change install_depexts to also pass in all the depexts in the switch https://github.com/ocaml/opam/pull/5982/files#diff-e8aa4752d602eeda81985e2ee34e48c4d6441eef00b51f5461c9cab855a0c0c3R1387 (which requires plumbing switch state through things), and we set old_packages to the difference between all_packages and new_packages.

I think we could perhaps create a new record to hold this 3-tuple, and the existing 2-tuple (which would have prevented this bug).

[kit-ty-kate]

Given that opam env isn't changed when a depext has been installed outside of a build (for good reasons, you don't want to have your environment replaced by another one just to get access to some tool), i'm wondering whether we should notify the users (when the use nix) that their current environment won't be modified, so that they are not surprised when they install conf-pkg-config or something and do not see it in their current environment.

For example if someone wants to work on a local project that use tools such as pkg-config, opam install . --deps-only won't be enough to get a working environment to build it locally, they would have to install system dependencies manually too.

So to sum up, the method chosen by this PR is great to install packages, but doesn't help when developing (which users arguably use opam more for). I think this is fine, but users definitely need to get notified when a depext is being installed.

[kit-ty-kate]

@RyanGibb regarding my last comment, is there a nix command that one could run to enter a clean new shell with an environment as if opam was building the package? Would nix-shell with one -p for each package that was installed be sufficient? (e.g. nix-shell -p pkg-config -p openssl -p gmp)

If this is reasonable, maybe we cold display it as a hint for users to run after an opam install --deps-only <local dir> build where the dependency tree contains depexts?

[RyanGibb]

Oh, good point! nix-shell -p pkg-config openssl gmp etc... should work (though requires a channel to be set up). An annoyance with this approach is packages from these ephemeral shells can be GC'd, breaking built artifacts.

[kit-ty-kate]

wouldn't these packages be part of the env.nix file and thus be listed in the Nix root thingy that would make them not GCable? (cf. the PR description)

[RyanGibb]

Ah, yes, you're correct. Assuming they haven't changed the channel, that should work quite nicely!

[kit-ty-kate]

Thanks a lot!

[RyanGibb]

Wow, awesome that this got merged at long last! Thank you both for all the work you put in :-)