Proposal to introduce plugins

[jmgilman]

I apologize for straying away from the template, but I think doing so makes this proposal a bit easier to read. I've been working on Nixago for the past couple of weeks with the intention of blossoming it into a larger project aimed at generating development environments with Nix. Given the overlap between this idea and devshell, I thought it prudent to first bring my proposal here to see if it might fit in with the vision of the maintainers.

The below proposal integrates most of the framework I've been developing for how these environments would be configured. I'm more than happy to take this conversation over to a more real-time format, please let me know if there's a Slack/Matrix/etc. space that I can join.

Proposal

This proposal considers using abstraction to create a user-friendly interface for further configuring a development environment. I recognize that this is a rather large change and may not be compatible with the author's original intentions. However, I do believe it enhances the underlying principle of improving development environments/workflows with Nix.

Plugins

The proposal is to add support for a concept called a plugin. A plugin is a Nix expression that takes a variable input and produces a standardized output. Contractually, devshell is not concerned about the input. However, the output of the plugin will be in such a format that can be interpreted and executed by devshell.

The primary purpose of a plugin is to control the startup behavior of a shell environment created by devshell. A plugin may modify any of the following:

• The environment variables available
• Shell aliases
• Available Nix packages
• Additional shell hooks to execute when the shell is created

Plugins are external to devshell, although providing first-class support for widely used ones may not be out of the picture. Developers create plugins using a standardized schema and make them available for others to use. Plugins are added to devshell through an interface that accepts a list of plugins. Recall that plugins are nothing more than Nix expressions that adhere to a contractual output schema: devshell evaluates these expressions and executes the output, thus modifying the resulting environment.

Example

A developer wants to integrate support for the just task runner into devshell. A plugin is written which takes the following input:

{
  vars = {
    myvar = "myvalue";
  };
  tasks = {
    mytask =
      [
        ''echo "Doing the thing with {{myvalue}}"''
        ''do_thing {{myvalue}}''
      ];
  };
}

The plugin returns an output that will be interpreted by devshell. The schema for this is TBD, but one can imagine the following:

{
  aliases = {
    j = "just"; # Alias `just` to `j`
  };
  packages = [ pkgs.just ]; # Add just to local environment
  shellHook = "<define hook here>" # Nixago creates a shell hook which generates and links the `.justfile`
}

The shellHook is where Nixago could potentially fit into the picture. When invoked, Nixago returns a derivation for building the configuration file and a shell hook that interpolates it (thus causing it to be built) and links it to the local environment. The resulting environment created by devshell would enable the j alias, add pkgs.just as an input to mkShell, and append the shellHook. The developer would then be able to run j mytask to execute the defined task.

This example could be extended to support several different tools. The result of combining one or more plugins is a customized environment that can be mix-and-matched to create something suitable for a project. For example, in my own environments, I largely use just for task running and lefthook for git hooks. My goal is to define simplified configurations for these tools and have it just work when the shell environment is created by Nix.

Nixago

Here is some additional information about what Nixago provides:

Nixago is a Nix flake library for generating configuration files. It primarily targets development environments by dynamically generating configuration files for common development tools. General usage involves defining the configuration data in the user's flake.nix and then adding the generated files to .gitingore. The result is a less cluttered repository root and creates a central source of truth (similar to the way Poetry consolidates with pyproject.toml).

See the docs for more information.

Conclusion

The proposal essentially accomplishes two things:

1. Greatly increases the ability to customize a devshell environment.
2. Allows easy integration of Nixago for generating configuration files

The result is that Nixago remains largely detached from devshell and primarily interacts with it through the shellHook output of the example schema. This seems desirable to me as I don't see the need for tighter integration. A plugin author could use any number of systems to generate an appropriate shell hook.

The plugin system, in general, is a large addition to devshell, but I think such systems are valuable for the longevity of a project. Maintaining a simple contract between plugin developers helps to increase the rate at which additional functionality can be added to devshell. Keeping in line with the README, one could imagine a team maintaining a set of plugins for standardizing their own development environments (i.e., we use just and lefthook for tasks/commit hooks).

Finally, I'm more than happy to spearhead this proposal. I've already started work on a project to implement it, but as I noted before, I have paused to consider adding the functionality to devshell instead of running a mostly parallel project. I would require support in the beginning to better wrap my head around the devshell architecture and how this proposal would fit in. Due to external circumstances, I am available to work on this integration full-time for the next few weeks.

Thank you for considering this proposal.

[blaggacao]

I may add, that I'm giving the integration between devshell and nixago a shot and this is my first conclusion:

https://github.com/nix-community/nixago/issues/29

[zimbatm]

Hi, thanks for writing this proposal! I don't have a lot of time available so sorry for the short reply.

I took a look at Nixago and have one big question; why use Cue instead of the NixOS module system? I think it makes sense to generate config files from JSON-like data structure and that's also something nixpkgs.formats does. If it was a module system, it could also potentially plug into devshell a bit more.

In either case, I think it would integrate pretty well.

devshell replaces the shellHook with a startup script, so you can use that instead. It's basically the same thing but is more composable: https://numtide.github.io/devshell/modules_schema.html#devshellstartupnamedeps

[jmgilman]

Thanks for the feedback!

Regarding CUE, it's only one of the engines that Nixago uses. The primary reason behind introducing it was because I felt that Nix was not really purpose-built like CUE to tackle this area of managing configuration files. I can write validation and transformation logic in CUE with much better feedback at a much faster rate than compared to Nix.

It should be possible to write a "module engine" for Nixago that meets the criteria of your suggestion. The main goal of Nixago was standardizing an interface for generating and managing configuration files that can easily be extended to use whatever tooling is already being used by a shop.

As far as integration goes, Nixago was really just an example. The proposal itself is more focused on the plugin architecture and not necessarily on where Nixago fits in.

[zimbatm]

I must be stupid. I don't really get what the interface would look like and what it would do :sweat_smile:

If the idea is to symlink generated config files into the repo's file structure, I think this is an error-prone operation that requires careful alignment of .gitingore. The best is to generate a wrapper around the script that points to the config file in the nix store. Eg: just would a wrapper for just /nix/store/.../Justfile (assuming you can override the current directory)

[jmgilman]

A few things:

1. Yes, there is some proneness to error when symlinking files locally. We already have a feature being worked on to automatically sync .gitignore to prevent inadvertent commits of symlinks to the repository. If a user so chooses, they can just opt to use the "copy" mode which manages a local copy of the file which can be checked into the repository.
2. Symlinking locally allows a user to quickly grok what the current configuration is, vice having to search for its location in the Nix store.
3. Not all CLI tools offer a --config option. I know this pain firsthand, and so the alternative proposal would not work in these cases without submitting something upstream.

Nixago provides one primary function, make, which takes a single attribute set that is defined in the request module. The request module defines all applicable options for generating a configuration file, including its content and how it should be managed locally.

Thus, the interface is fairly basic. Define some options, pass them into make, and receive a derivation and shell hook. How this integrates into devshell is the origin of this proposal. The interface being proposed is a plugin system that Nixago can easily integrate into. Namely, devshell can receive an arbitrary number of plugins (which are nothing more than functions) that return a standardized response, one of which is adding additional shell startup logic (this is what Nixago would use to generate the configuration files).

I want to reiterate, though, that Nixago really isn't the primary candidate being proposed here. It's just background context on how the proposal came to be in the first place. The plugin approach sits on an abstract layer far above Nixago. I had intended on developing a project that paralleled devshell but utilized this plugin approach as a first-class citizen. Before going down that route, though, I want to bring the proposal here first to see how it might integrate into devshell instead of being put into another parallel project.

Hopefully, that clears things up a bit more.

[zimbatm]

I think it does, thanks.

How it would work with devshell would be to have these three layers:

1. Some mechanism like shellHook that is executed on shell entry. This already exists by setting devshell.startup..text
2. A "files" module where file paths can be mapped to content. Eg: files."Justfile" = "some content". That module would use the startup script to symlink files on startup. It would also have to be able to remove old symlinks somehow.
3. A "config" module that maps JSON-like data structures to "files".

(1) is already in devshell. Some variation of (2) and (3) exist outside of devshell in https://github.com/cruel-intentions/devshell-files . That's the beauty of the module system, it's easy to extend on your own.