dream2nix fully based on nixos modules?

[DavHau]

related: https://github.com/nix-community/dream2nix/pull/155 https://github.com/nix-community/dream2nix/pull/151 https://github.com/nix-community/dream2nix/pull/153

This is a hypothetical usage example of dream2nix if the framework was fully based on the nixos module system.

This is not only useful for configuring nixos systems. Even if the intention is to just create single packages or a dev shell, dream2nix could still be invoked via a module definition and benefit from properties of the nixos module system.

Benefits I see:

• Discoverable packages and package options (all derivation inputs configurable)
• Plugin interface. Translators, Builders, etc. can be added by setting options
• Simply make many other parts of the framework configurable via options.
• Type checked interfaces

dream2nix-ripgrep.nix:

{

  imports = [
    (builtins.getFlake "github:nix-community/dream2nix").module
  ];

  # cache directory for dream-lock files
  dream2nix.packagesDir = ./dream2nix-packages;

  # extend the framework with a new translator
  drem2nix.translators.some-new-translator = ...;

  # define projects for which to generate packages
  /*
    interface like:
    dream2nix.projects.<name> = {};

    As a result, top level packages of that project will be exposed via:
    `dream2nix.projects.<name>.packages`

    And dependencies can be overridden/configured via
    `dream2nix.projects.<name>.dependencies.<depName>.`
  */
  dream2nix.projects.ripgrep = {
    src = builtins.fetchGit ...;
    settings = [
      {
        builder = "crane";
        translator = "some-new-translator";
      }
    ];
  };

  # configure package on a per project basis
  # replaces `packageOverrides`
  dream2nix.projects.ripgrep.packages.grep = {
    preBuild = ''
      echo "user defined override for grep"
    '';
  };

  # global package overrides
  # (will usually be shipped by dream2nix upstream)
  dream2nix.overrides.rust.grep.fix-build = {
    _condition = ...;
    preBuild = ''
      echo "default dream2nix override for grep"
    '';
  };

  # For example, if the intention is to configure a nixos system,
  # one could just add packages from dream2nix to the systemPackages like that
  environment.systemPackages = [
    # add the ripgrep package from dream2nix
    config.dream2nix.projects.ripgrep.packages.default
    # add the grep package as well
    config.dream2nix.projects.ripgrep.dependencies.grep.package
  ];
}

For usage with simple nix expressions, outside of a nixos configuration, a utility function could be defined, taking care of calling evalModules correctly. This for example could be used in a flake: (./dream2nix-ripgrep.nix corresponds to the file above)

  let
    evaluated = callDream2nix ./dream2nix-module.nix;
  in
    evaluated.projects.ripgrep.packages.default;

@yusdacra @jaen

[jaen]

I would have to stew a bit on it and maybe a PoC would be useful to iron out the interface before committing to it, but certainly one thing you sketched here that I was thinking about was project configuration also being incorporated into the module system, not only the subsystem themselves. It seems like a good idea, and will probably make it easier to refer to projects without threading stuff around.

That said, I wouldn't forego namespacing of subsystems, eg. I would imagine it should be something like dream2nix.subsystems.some-language.translators.some-new-translator or dream2nix.translators.some-language.some-new-translator or something else to that effect. I probably would prefer namespacing things under dream2nix.subsystems but YMMV.

I see you're already doing something similar that with dream2nix.overrides.rust (unless I'm misunderstanding something, I think this is not the case now, so when packages from different languages are named the same in a project, you can't specify different overrides and injects). Also incidentally, shouldn't that be a list? The _condition part would probably handle different languages, but if it's an attrset, not a list, then you would only ever be able to have one.

Again, sorry if this is disjointed.

[jaen]

I've also only just noticed @yusdacra's #155 – while I think an interface similar to what you describe here looks like a nice end-state to be in, I think it might be useful to first have something like #155 done, so it can enable external extensibility sooner and give more time to figure out what exactly the module-based interface should look like.

[yusdacra]

I don't think I like using nixos modules for everything, especially for the user interface. While I think it would be okay for builders / translators etc. and overrides, I think we should keep the other interfaces as-is and not use modules. I think it would make it harder to integrate with other stuff in general, and feels too complex instead of just having validation IMO (maybe with something like https://github.com/divnix/yants). So it seems like it doesn't provide much value as opposed to the complexity it would bring.

Personally I think it would be best if this module system wasn't directly integrated into dream2nix, but rather built on top of it, as to keep dream2nix itself as simple as possible. dream2nix itself should expose primitives for making this possible, which then everyone can benefit from, and this would allow for people to use stuff other than nixos modules.

About #155, I think I like the approach there, and I think it could be even simpler (which I'll work on). I'm of the opinion that the less abstractions we have, the better we can read and debug the code, and nixos modules feel like the opposite of that, while not providing much value. The interface for extending can be changed there and I'm open to any ideas for that (the first change I'll do is probably just taking a list of files instead of subsystem, name, file attrset because the module should already have these).

[DavHau]

I see you're already doing something similar that with dream2nix.overrides.rust (unless I'm misunderstanding something, I think this is not the case now, so when packages from different languages are named the same in a project, you can't specify different overrides and injects). Also incidentally, shouldn't that be a list? The _condition part would probably handle different languages, but if it's an attrset, not a list, then you would only ever be able to have one.

Good point. Because a project currently is defined by a source tree and that source tree can contain sub-projects of different subsystems, we should make sure to categorize the project specific overrides and injects by the subsystem.

Concerning attrset vs list: The reason I chose an attrset for the global overrides is that each override will now have a key. This allows us to build some more advanced debugging features later, where the user can display a list of applied overrides or can exclude overrides etc. With this model there can already exist many overrides for a single package. Therefore I see no benefit in using a list.

I don't think I like using nixos modules for everything, especially for the user interface. While I think it would be okay for builders / translators etc. and overrides, I think we should keep the other interfaces as-is and not use modules. I think it would make it harder to integrate with other stuff in general, and feels too complex instead of just having validation IMO

Why would it make the user interface harder to integrate with other stuff? Currently our interface consists of functions which receive an attrset. A nixos module can be an attrset as well. So there wouldn't be a reason why the user interface should look any different by default. We can make the user interface however we want it. Users wouldn't have to know that the function argument is interpreted as a nixos module.

Right now there are a lot of uncertainties in our user interface. We have overrides, sourceOverrides. we have the dream2nix config, we have settings, for all of which it is not clear by what logic those are applied. Does my setting break an existing setting? Are lists merged or replaced? Does my override conflict with an existing one? Those are all questions where we need to find a consistent logic, which is something the nixos module system already has in place and its mechanisms are well understood by the community.

So it seems like it doesn't provide much value as opposed to the complexity it would bring

I feel it is going to be the opposite way. It would reduce the complexity of our code if we would just re-use the existing module system. Right now and especially after #155 A lot of our code is just about extensibility, how we load our own modules, how extra modules can be loaded, how settings are applied etc. All of that will simply vanish if we use the nixos module system. Currently we are about to re-implement our own module system and we are probably going to re-do all the mistakes that have already been solved by nixos modules. Why not just re-use what is already there and is maintained by the community. Less work for us.

But I agree, that making the whole framework module based at once might be too big of a step. First we should start using nixos modules where can really benefit (overrides, settings, config, etc.) and see how this works out. I think I might start working on a POC with something simple like the fetchers and see if my hypothesis of the code becoming less complex turns out to be true.

[yusdacra]

I agree that nixos modules would give more discoverability and structure which is why I think it makes most sense for overrides (source, package, injecting dependencies too potentially) since it's the most user facing interface. But for builders / translators etc. I think they are really simple in nature (pretty much only type and translate / build) and don't really need nixos modules to be used. They also don't really need to be discoverable (aside from knowing which translators / builders are available). I thought more upon that and the statements before from me about what should use nixos modules and what should not don't really make sense now.

I'd like to keep dream2nix's core stuff ideally depending on as less nixpkgs code as possible so maybe one day we can make it portable... So I think instead of integrating nixos modules into dream2nix, we can build on top of it so that it's not a strict requirement (eg. you could skip using it and directly use the underlying interfaces). Even if it is maintained by not us, it still needs to be maintained (meaning eventually we might need to touch it), and it still means we depend on more code (lib/modules.nix in nixpkgs is over 1k lines). So it would be good to have an "escape hatch" just in case. We would keep the core simple and dependent on as less stuff as possible while building everything else on top of that while integrating with other stuff (eg. nixos modules, yants).

[infinisil]

At Zurihac I developed a small demo for using the module system for dream2nix with the feedback of @DavHau, here's the result, mainly see the user.nix module, which is the user interface: https://github.com/infinisil/dream2nix-modules

[DavHau]

At Zurihac I developed a small demo for using the module system for dream2nix with the feedback of @DavHau, here's the result, mainly see the user.nix module, which is the user interface: https://github.com/infinisil/dream2nix-modules

Thanks a lot for this. To add some information, the scope of the demo is to replace the current packageOverrides, not to make the full framework module based.

[yusdacra]

I took a better look at the POC and this could be nice, but I'm not sure if I understand it right. My understanding is that the packages is for creating packages? And overrides is basically what packageOverrides currently does, but you can set it per builder. Is that correct?

Based on my understanding I feel like we could make it better if we:

• changed packages to have a structure like packages.${name}.${version} so that you can create packages with different versions.
• changed overrides to have a structure like overrides.${subsystem}.${builder}.${overrideName}, to avoid name collisions and to have a more consistent interface for overriding packages. Basically:

{
  overrides.packages.rust.crane.my-override = {
    # attribute for deciding which packages to attribute with this
    # you could also set it to `_: true` to override all packages
    # `pkg` should be an attribute set containing a `name` and a `version`
    # so you can select both at the same time.
    enableIf = pkg: lib.any (n: pkg.name == n) ["ripgrep" "hello"];
    # or for example
    enableIf = pkg: pkg.name == "ripgrep" && (builtins.compareVersions pkg.version "13.0.0" >= 0);
    # you can also set a simple attribute to enable it for one package only
    enableForPackage = "ripgrep";
    # set builder config / arguments
    config = {
      runTests = false;
      profile = "debug";
      # builders can expose attributes like `overrideAttrs` for overriding derivations directly
      overrideAttrs = old: {
        MY_ENV_VAR = "my value";
      };
      # since this is an example for the `crane` builder, it can also expose something like this
      # (since it builds two separate derivations, one for deps-only and other for main package)
      overrideDepsAttrs = old: {
        someEnvVarForADependency = "some value";
      };
    };
  };
}

• added support for source overrides, maybe like so:

{
  overrides.sources.my-override = {
    # pkg contains a `name` and `version` like `overrides.packages`
    # pretty much the same as that
    enableIf = pkg: pkg.name == "ripgrep" && pkg.version == "13.0.0";
    # we can add helper functions like this to make it easier
    enableIf = dlib.matchNameVersion "ripgrep" "13.0.0";
    changeSource = old: /* patch the old source or something */;
  };
}

• and dependency injections:

{
  overrides.dependencies.my-override = {
    # you get the gist :P
    enableIf = pkg: pkg.name == "ripgrep" && pkg.version == "13.0.0";
    changeDependencies = old: old ++ [
      ["foo" "0.1.0"]
      ["bar" "0.2.0"]
    ];
  };
}

I think doing them like this would make the interface way more consistent than how it is currently, which I think is a good thing; instead of all the different interfaces you now have one entrypoint, and similar interfaces for the packages / sources / dependencies. We could still add helpers to make it even easier like so:

{
  overrides.sources.my-override = dlib.overrides.sources.mkSingle {
    name = "ripgrep";
    version = "13.0.0";
    source = old: /* something */;
  };
  # or maybe
  overrides.sources.my-override = dlib.overrides.sources.mkSingle "ripgrep" "13.0.0" (old: /* something */);
  # or maybe even (by-name-version is reserved)
  overrides.sources.by-name-version.ripgrep."13.0.0" = old: /* something */;
}

that is up to discussion, and I'm not sure what would work better or not.

[DavHau]

My understanding is that the packages is for creating packages? And overrides is basically what packageOverrides currently does, but you can set it per builder. Is that correct?

Correct.

changed packages to have a structure like packages.${name}.${version} so that you can create packages with different versions.

I agree. There are still some things that need to be re-structured.

changed overrides to have a structure like overrides.${subsystem}.${builder}.${overrideName}, to avoid name collisions and to have a more consistent interface for overriding packages. Basically

Concerning the enableIf. I think that the problem with that is that all overrides have to be checked against all packages. If we have 1k dependencies and 1k overrides, we'd have to do a million checks. Therefore I think we should be namespacing overrides by the package's name. That allows us to reduce the number of overrides we need to check the condition for drastically.

I'm not sure if overrides should be namespaced by builder. Within a subsystem, there will be options which are shared between all builders and there will be options which are specific to a builder. I'm concerned about the community overrides. Most of the options we set there will probably be options which apply to all builders. If we would namespace overrides by builder, we'd have to duplicate many overrides whenever there are multiple builders. Maybe it's better to not namespace by builder and instead check for the builder in the condition (ea. enableIf) if necessary.

I'm thinking of something like:

overrides.packages.rust.ripgrep.my-override = {
  # maybe we should not put the whole `pkg` as an input here for now?
  enableIf = {builder, version, ...}: version == "1.2.3" && builder = "crane";
}

Though the drawback of this would be that we won't be able to typecheck overrides before they are actually applied, because we don't know which builders interface they apply to.

Maybe an alternative to this could be to have a base builder for each subsystem containing all the generic options and all other builders would just extend that interface. Then we could structure the overrides like: rust.ripgrep.base or rust.ripgrep.crane etc...

Concerning the options for sources and dependencies, I agree that this should be included as well. The result will basically be that the make[Flake]Outputs function will receive a module/config as an argument. But it might be best to transition step by step and start with packageOverrides first?

[yusdacra]

changed overrides to have a structure like overrides.subsystem.{builder}.${overrideName}, to avoid name collisions and to have a more consistent interface for overriding packages. Basically

Concerning the enableIf. I think that the problem with that is that all overrides have to be checked against all packages. If we have 1k dependencies and 1k overrides, we'd have to do a million checks. Therefore I think we should be namespacing overrides by the package's name. That allows us to reduce the number of overrides we need to check the condition for drastically.

I think if we have the enableForPackage option that I mentioned, we could simply just build an attrset without evaling anything using that (since we know the keys beforehand), with builtins.listToAttrs. For everything else we would still need to eval, but it would be a minority so I think it would be okay.

I'm not sure if overrides should be namespaced by builder. Within a subsystem, there will be options which are shared between all builders and there will be options which are specific to a builder. I'm concerned about the community overrides. Most of the options we set there will probably be options which apply to all builders. If we would namespace overrides by builder, we'd have to duplicate many overrides whenever there are multiple builders. Maybe it's better to not namespace by builder and instead check for the builder in the condition (ea. enableIf) if necessary.

I'm thinking of something like:

overrides.packages.rust.ripgrep.my-override = {
  # maybe we should not put the whole `pkg` as an input here for now?
  enableIf = {builder, version, ...}: version == "1.2.3" && builder = "crane";
}

Though the drawback of this would be that we won't be able to typecheck overrides before they are actually applied, because we don't know which builders interface they apply to.

Maybe an alternative to this could be to have a base builder for each subsystem containing all the generic options and all other builders would just extend that interface. Then we could structure the overrides like: rust.ripgrep.base or rust.ripgrep.crane etc...

I think we should have it be namespaced by builder but have a base builder as you said. So overrides.packages.rust.base for common overrides that apply to all packages regardless of builder. We could also have a overrides.packages.all that apply to every package regardless of subsystem / builder, maybe useful.

Concerning the options for sources and dependencies, I agree that this should be included as well. The result will basically be that the make[Flake]Outputs function will receive a module/config as an argument. But it might be best to transition step by step and start with packageOverrides first?

Yes, I think that's okay. To make it easier to debug packages and incrementally move to modules, I think it would be okay if we took an argument like _module in make[Flake]Outputs, and then as we add things to that we can remove the other arguments, and eventually replace the whole argument to the function with a module.

[wmertens]

I think the idea is very straightforward, and it's only the details that need some thought.

Summary as I understand it:

• dream2nix adds dream2nix to config. In the absence of nixos, config is just an empty object. I'll shorten config.dream2nix as d2n below
• d2n holds only very generic settings, and everything is in submodules, so it can be typechecked.
• d2n.projects.${name} holds per-project configuration. This is the level of the current flake.nix configuration, probably it should be passed around as project at that level
• currently, translators are defined under d2n.translators and they specify their subsystem name

A point of some confusion for me at first was that e.g. translators are both configuration and executable code that takes configuration. Of course, for Nix it's all just lazy evaluation.

I'd like to propose these changes:

• translators are not first-level, but are stored under their subsystem
  • (.subsystem is so long, how about .lang?)
• so you'd have e.g. d2n.subsystems.nodejs.translators.package-lock, typechecked by the module code
  • perhaps each translator should have a discover function, which determines if it can work with the given workspace tree (+ project tree). That way they are pluggable and discover doesn't need to know about them. The first one to return true wins, and you can configure the order
  • each translator, discoverer, builder etc that gets called, get d2n as an argument so they can read configuration.
• subsystem configuration, like which version to use, goes under d2n.subsystems.config, but modules allow e.g. translators to add their own configuration so e.g. d2n.subsystems.nodejs.translators.package-lock.patchLock
• d2n.packages combines all packages of all projects

Using this modular approach also allows having multiple subsystems within a single project, building different packages. It also allows adding entire new subsystems.

[wmertens]

And as for overrides, there's several places where it might be useful to intervene. During translation, after dream.lock generation and then during build.

Right now only the latter is possible, and it would be nice to have a single configuration object for all types.