[RFC 0083] Common interface package sets

[FRidh]

rendered

[FRidh]

cc @adisbladis @doronbehar @jtojnar @Ericson2314 because of your various involvements with package sets.

[shlevy]

Possible interaction with flakes: I'd like to be able to have the functions to build a package set in a flake, as well as the ability to declaratively overlay package sets in my flake config.

[Ma27]

Even though I like flakes, I'm not so sure if it's a good idea to design any RFC around any feature that's not yet stabilized (i.e. the APIs can all change and it's not in a stable Nix release) or even documented.

[fare]

In some package sets, you might want to combine multiple aspects: which version of the compiler should I use with which version of the libraries?

Plus some libraries are only available on some versions of the compiler, and I may want to pick and choose: "oh, I'd like these stable libraries with version 3.4 of the compiler, except I want override these particular libraries with newer versions I need—plus their dependencies."

I don't think the job of nixpkgs would be to solve complex sets of dependency constraints, but I believe its job should be to check and enforce the declared constraints (unless explicitly lifted / overridden by the user).

For all these reasons, I think it would be nicer to standardize on some extension mechanism when there are so many incompatible variants of the same thing.

At some point, I would offer POP as a better such extension mechanism to standardize on, that allows for a DAG of dependencies between aspects of your package set. But POP is not quite ready for that yet (the API needs to be made nicer, and I ought to use the C3 algorithm for linearization). [See indeed #82 for a proposal that allows to introduce POP smoothly into nixpkgs, by first tagging it as experimental.]

[fare]

Also, another thing I'm trying to do is allow override of "source" settings before libraries are turned into derivations. Either there should be a standard for such settings to be included in passthru, or for otherwise organizing collections of pre-packages that allow such overrides.

[bb010g]

Something I don't see this RFC addressing is a common way to overlay all instantiations' package sets at once. Using Python as an example, this is similar to patching the source of pkgs/top-level/python-packages.nix. To ultimately instantiate a Python package set pkgs.python37.pkgs, first pkgs.pythonInterpreters.python37 is evaluated, instantiating pkgs/top-level/python-packages.nix with that python to provide the .pkgs package scope. pythonInterpreters, a common parent for all instantiations, would be a natural place to allow for pan-instantiation customization.

An overlay evaluation chain could look like:

• (for consistency with lib.makeScopeWithSplicing moving forward, let lib.makeScope' be lib.makeScope but with proper .overrideScope)
• pkgs.fooVariants = pkgs.callPackage ../foo/default.nix { };, meaning overlaying pkgs reevaluates pkgs.fooVariants due to pkgs.callPackage
• Together:
  • pkgs.fooVariants.pkgsOverlay = import ../top-level/foo-packages.nix;
  • pkgs.fooVariants internal mkFooPkgs = (lib.makeScope' pkgs.newScope (fooPkgs: { foo = fooBase; }).overrideScope pkgs.fooVariants.pkgsOverlay, meaning overlaying pkgs.fooVariants.packages reevaluates all fooPkgs instantiations
  • (this example has foo-packages.nix handle turning fooPkgsSuper.foo into a properly integrated fooPkgs.foo)
• pkgs.fooVariants.fooBase_a = callPackage ./foo-a.nix { };, with normal (pkgs) scope-provided callPackage
• pkgs.fooVariants.foo_a = prepareFoo (mkFooPkgs pkgs.fooVariants.fooBase_a);, where prepareFoo finishes setting up a top level foo from fooPkgs containing fooPkgs.foo, meaning overlaying pkgs.fooVariants.foo_a.pkgs reevaluates pkgs.fooVariants.foo_a, but not other fooVariants.foo evaluations
• pkgs.foo_a = pkgs.fooVariants.foo_a;
• pkgs.foo = pkgs.foo_a;

It may be desirable to make pkgs.fooVariants a package set to avoid having to access the fix-point fooVariants through pkgs internally. This would turn pan-instantiation overlays at the pkgs level into something like

pkgs: pkgsSuper: {
  fooVariants = pkgsSuper.fooVariants.overridePackages (fooVariants: fooVariantsSuper: {
    pkgsOverlay = fooVariantsSuper.packages.overridePackages (fooPkgs: fooPkgsSuper: {
      newPkg = fooPkgs.callPackage … { };
    });
  });
}

. A helper function fooVariants.overridePkgs could simplify this to

pkgs: pkgsSuper: {
  fooVariants = pkgsSuper.fooVariants.overridePkgs (fooPkgs: fooPkgsSuper: {
    newPkg = fooPkgs.callPackage … { };
  });
}

, where

self: super: { overridePkgs =
  pkgsOverlay: self.overridePackages (self': super': {
    pkgsOverlay = lib.extends pkgsOverlay super'.pkgsOverlay;
  })
; }

.

I think pkgsOverlay is a pretty reasonable extension point for cases like NUR overlays that want to add first-class packages to package sets. These sorts of overlays shouldn't be brittle based on what fooVariants were present last time you checked. Your overlaid packages should float to new variants naturally, along with the rest of normal foo.pkgs.

[fare]

I updated the RFC with a comparison to the module system as adapted from https://github.com/NixOS/nixpkgs/pull/116275#issuecomment-879142249

I also indicated that my paper Prototypes: Object-Orientation, Functionally was accepted at the Scheme and Functional Programming Workshop 2021. The version that was accepted is significantly improved since earlier drafts from when I first created this RFC. The paper explores the foundations of object-oriented programming, and rebuilds (in Scheme) the model used by POP. It has an extended "Related Work" section that of course mentions previous Nix extension systems, but goes back to the 1970s, and a lot of things in between.

[Profpatsch]

Having a published paper at the Scheme conf really piked my interest in this, I’ll try to read the paper when I find the time.

Nixos does have a quite big sunk cost on its current module system, but e.g. https://github.com/tweag/nickel/ might be a good candidate for applying this research! cc @yannham