svanderburg
Partial regenerations
Running node2nix for a large number of packages, like in nixpkgs itself, can take quite a long time.
It would be nice to be able to do a partial regeneration, to make this quicker (and perhaps reduce merge conflicts).
I can see roughly 2 ways to do this:
- Introducing an 'amend mode' which instead of generating a
node-packages.nixfrom scratch, takes an existingnode-packages.nixand adds newly-discovered dependencies to it. - Instead of having everything in one big
node-packages.nix, generate an intermediate file per dependency, and somehow merge those.
The first approach would require a way to parse .nix expressions - AFAICS https://github.com/svanderburg/nijs doesn't support that yet, right?
The second approach would be a more invasive change, but perhaps also ultimately more elegant? Should the intermediate files be .nix expressions or .json files? Perhaps they could be a JSON representation of the CollectionExpression for each dependency, and then when one of those is updated node2nix could collect all those intermediate CollectionExpressions, merge them and save the result in .nix format?
@raboof Another approach could also be to generate and use a package-lock.json or something similar to it.
I'm currently investigating options.
Indeed, one of the options would be to read Nix expressions from node2nix, but (as far as I know) there is no JavaScript-based infrastructure yet that can do this -- NiJS' purpose is not to parse Nix expressions (to a JavaScript AST-like structure) but the opposite -- its main purpose is to generate Nix expressions from JavaScript code constructs (e.g. strings, arrays, objects) and domain models implemented in the JavaScript language.
A couple of years ago, when I modified NiJS to more easily facilitate transformations of custom objects to Nix expressions, I have also developed an incomplete feature (that still resides somewhere on my hard drive) that could serialize an object structure of NixObjects to JSON, so that you could save it to disk and consume it a later point for future modifications.
In theory, I could resurrect this experimental/incomplete serialization feature, finish it, and use that to save a symbolic representation of a generated Nix expression in JSON format. This could serve as a cache mechanism, and can be implemented quite quickly.
The downside, however, is that we need to save and commit two kinds of generated files in the Nixpkgs repository -- the Nix expressions and the corresponding JSON serialization. As you may probably already know, the churn for each commit is quite high, and this will double if we need to update two files.
The suggestion of @MetaDark is also something that is also on my radar -- one the things I'm working on a companion tool that can more easily perform all required modifications on a node_modules hierarchy. The data exchange format is very similar to a package lock file. We might also potentially use these files to facilitate partial regenerations.
I have also been thinking about the hypothetical scenario in which we could parse Nix expressions -- this would still lead to impracticality that Nix expressions do not capture data, but also build instructions and function abstractions. I don't think it's very practical to consume such specifications. It probably also introduces additional challenges if we decide to change the format.
btw. the unfinished JSON serialization feature of NiJS that I just described is pretty cool IMO. Originally, I have developed it to help me debugging applications that use NiJS for data exchange more easily (these applications were mainly company-specific, not node2nix).
Sadly, due to time constraints and other obligations, I never fully finished it. Because I did not have any other use case, I more or less put that idea on hold.
If people do not have any objections against doubling the commit churn in Nixpkgs, then this could also already be a solution for supporting partial regenerations that I could implement in a relatively short amount of time. I still think that if you do a comparison, the total amount of churn will be lower because we only have to make modifications for the dependencies of one single package, not all packages.
A solution like this will keep node2nix on a trajectory to solve Nixpkgs' problems well, whereas otherwise, the consensus may tend towards switching to a fixed-output-derivation method of fetching, which is very much a local optimum and not one we want to be stuck in.
