Make it possible to build NPM sub projects in Monorepos

[svanderburg]

I have observed that in addition to deploying NPM packages from registry and pure NPM development projects, it has also become quite common to deploy Monorepos with NPM sub projects, or hybrid projects (e.g. a Python project with NPM dependencies).

Currently, there is no convenient way to generate Nix expressions for embedded NPM projects and integrate the corresponding deployment procedure into the Nix builder environment of the parent project.

[stephank]

I've been thinking about this for a bit, coming from Yarn workspaces. I've tried to skim the node2nix code before, but don't yet have a good understanding, so hopefully I'm not rambling too much. 🙃

Simple example setup:

• package.json:

{
  "private": true,
  "workspaces": ["a", "b"]
}

• a/package.json:

{
  "name": "a",
  "version": "0.0.0",
  "scripts": {
    "prepare": "tsc"
  },
  "dependencies": {
    "express": "4"
  },
  "devDependencies": {
    "typescript": "3.9"
  }
}

• b/package.json:

{
  "name": "b",
  "version": "0.0.0",
  "dependencies": {
    "a": "*"
  }
}

Thoughts:

• Node2nix can mostly generate a and b individually, I believe? As long as the generated derivations for dependencies are similar, cache should be shared and any duplication is not a big issue. A complete understanding of workspaces / monorepos in node2nix might not be necessary for, let's say, phase 1.
• b depends on a, but a must not be resolved through NPM, plus a needs to be built first. Solving this is the minimum, I think.
  • There already exist path dependencies ("a": "../a"), but I don't think they solve our problems. They simply generate src = ../a buried deep, which is difficult to override with a custom derivation, for example.
  • Without path dependencies (as in the example above, "a": "*"), node2nix needs logic to redirect resolution of a to a/package.json (instead of the NPM registry) at generation-time. The dependencies of a should be available to b, but not the dev dependencies. Perhaps we can allow the user to indicate workspace dependencies with CLI flags or similar to --supplement-input?
  • To wire b to a custom derivation for a, maybe node2nix can generate a Nix factory function for when workspace dependencies are used, so it's up to the user to provide a?
    • Thinking this flexibility may be useful to support different monorepo / workspace solutions.
    • This also allows flexibility to wire a to a repo-wide build step (like a TypeScript workspace / tsc -b).
• Having these basics in place will allow a phase 2 to make all of this more user-friendly:
  • Parse workspaces, run generator for all subpackages, generate Nix expression to wire b to tarball attribute of a.
  • Shared node-env.nix in the toplevel directory.
  • Shared sources list. These are currently tucked away in node-packages.nix. Could move buildNodePackage stuff to the default.nix files for (sub)packages, or combine all (sub)packages in a single top-level node-packages.nix. (What this looks like may also depend on your node-env refactor.)
• Non-Node.js projects in the same monorepo / other monorepo styles can hopefully build on phase 1 stuff to do similar work.
• Additional difficulty: The top-level Yarn workspace package.json may also specify dependencies. These may be necessary for a project-wide build, like tsc -b. (This may not even be specific to Yarn, but apply to any kind of project-wide build step.)

[stephank]

The very last point is interesting, and is also something I came up against in the PoC. There are two possible contexts for node2nix here:

1. Subpackage context, when run against a subpackage package.json. This is intended for deploying the subpackage by itself, or perhaps as a (isolated) build step for an internal package. In this case, workspace interdependencies should be bundled from output of a custom derivation.

2. Workspace context, when run against the top-level workspace package.json. This is intended for deploying the workspace as a whole, or perhaps as an intermediate needed for a repo-wide build step. In this case, workspace interdependencies should be symlinked in node_modules, and subpackage devDependencies may also need to be available.

Separately, I'm thinking it could be powerful to have a generic mechanism to override any src from Nix. Then we can treat workspace dependencies more like local dependencies, which also offers a convenient default if there is no build step. (But point 2 is still special because of the symlinks.)

[evanjs]

Pretty sure I encountered this lately when trying to package hyve, which has multiple sub projects, but also depends on @vue/cli-service, which is itself a subproject of @vue/cli.

I would imagine this is the reason I encountered some strange errors when attempting to parse versions of dependencies for @vue/cli-service.

For example:

http request GET https://registry.npmjs.org/vue-loader-v16
http 200 https://registry.npmjs.org/vue-loader-v16
Cannot resolve version: vue-loader-v16@undefined

Going to put this on hold until this is figured out, as it seems impractical to try to get this working if node2nix can't handle it :D

[rien]

What is currently the most convenient way to package yarn workspaces?

I have a monorepo with a few (5) projects where it is possible to work out the internal dependencies by hand. I was thinking of building the core library and then using the resulting derivation to "inject" within the other packages. What would be the most straightforward way to do this?

[K900]

Any news here? It's currently blocking updating Prometheus in nixpkgs...

[svanderburg]

I'm still working on a better approach that allows you to perform a fake npm install from any script, but that is still a work in progress and not an easy problem to solve in the current architecture of node2nix.

In the meantime we should probably look for a workaround...

[K900]

Do you have that work published anywhere? We could test it on Prometheus if nothing else...

[mirtyl-wacdec]

Hi, has there been any progress in this?