floco
floco copied to clipboard
aakropotkin
Using Nix to put NPM and Yarn in a coffin
#+TITLE: floco
=floco= is a JavaScript package management and build tool powered by [[https://nixos.org][Nix]].
=floco= is a bold departure from conventional JavaScript package management tooling focusing on reproducibility, distributed caching, and sandboxing.
Every package is built in a strictly declared sandbox isolated from the runtime system, much like an /OCI/ container. This approach allows packages to be built once and reused on any system that shares the same architecture and platform. Artifacts are cached locally and may be distributed among a cluster of systems allowing development environments to be created at more than twice the speed of =npm= or =yarn=.
Underlying installers are implemented in =bash= and limit themselves to using =coreutils=, =findutils=, and =jq= improving reproducibility and readability compared to tooling implemented using a sprawling maze of JavaScript.
Despite this repository's seemingly small form, it is the result of nearly a year of exploration, trial, and refinement. A great deal of effort was expended to make this piece of software /suck less/ than the competition, but rest assured that these routines were not implemented naively. Writing a JavaScript package management framework in =bash= and =nix= was done /not because it is easy/ - on the contrary it was fucking hard.
- Getting Started
There's a dedicated [[https://github.com/aakropotkin/floco/blob/main/doc/guides/basics.org][Getting Started]] guide that is the best place to dive in.
For projects that depend only on registry packages without =install= scripts that require native dependencies or cycle breaking, the following process will have you up and running: #+BEGIN_SRC shell set -euo pipefail; cd ./my-project; nix flake init -t github:aakropotkin/floco; nix run github:aakropotkin/floco -- translate; [[ -r ./package-lock.json~ ]] && mv ./package-lock.json{~,}; nix build -f ./. global; ls -R ./result; #+END_SRC
** Documentation
A collection of documentation is hosted on GitHub on the project's
[[https://github.com/aakropotkin/floco/wiki][wiki]] tab, and is also
available under
[[https://github.com/aakropotkin/floco/blob/main/doc][
Additionally all CLI tooling and scripts support the =--help= option. The =floco help CMD= sub-command may also be used to view "help" messages for a given =CMD=.
** Templates
A simple template with the boilerplate needed for use with our /updaters/ is available through ~nix flake init -t github:aakropotkin/floco~.
More templates are on the way.
- CLI
The =floco= CLI interface is currently under active development and is expected to change rapidly in the near future.
** Common Behaviors
Across the =floco= CLI a few behaviors are consistent.
The following config files are included/applied if they exist:
- ~/etc/floco/floco-cfg.{nix,json}~
- =${XDG_CONFIG_HOME:-$HOME/.config}/floco/floco-config.{nix,json}=
- "Local" =floco-config.{nix,json}= searched for between =PWD= and git project root, or =/= if =PWD= is not a =git= repository checkout.
References to =floco= will be pulled from the nearest =flake.lock=, or =nix registry list=, using =github:aakropotkin/floco/main= as a fallback.
** =floco translate=
Generate a =pdefs.nix= file from a =package[-lock].json= or registry package.
This routine utilized =npm= internally to resolve packages and form =node_modules= trees.
*** Local Project Example
#+BEGIN_SRC shell mkdir -p /tmp/foo; pushd /tmp/foo; echo '{ "name": "@floco/phony", "version": "4.2.0", "dependencies": { "lodash": "^4.17.21" }, "scripts": { "build": "touch ./built" } }' > ./package.json; nix shell github:aakropotkin/floco; floco -- translate -pt; nix flake init github:aakropotkin/floco -t; floco build; # or `nix build -f ./. global; ls ./result/lib/node_modules/@floco/phony/; #+END_SRC
*** Remote Project Example
#+BEGIN_SRC shell mkdir -p /tmp/foo; pushd /tmp/foo; nix shell github:aakropotkin/floco; floco -- translate -pt [email protected]; nix flake init github:aakropotkin/floco -t registry; echo '{ ident = "lodash"; version = "4.17.21"; }' > ./info.nix; floco build; # or `nix build -f ./.;' ls ./result/lib/node_modules/lodash/; #+END_SRC
** =floco list=
List all declared projects by "key", being =<IDENT>/<VERSION>=, such as =@foo/bar/4.2.0= or =baz/4.2.0=.
#+BEGIN_SRC shell :exports both :results output nix run github:aakropotkin/floco -- list; #+END_SRC
#+RESULTS: #+begin_example @webassemblyjs/wast-printer/1.9.0 @xtuc/ieee754/1.2.0 @xtuc/long/4.2.2 abab/2.0.6 abbrev/1.1.1 abbrev/2.0.0 abort-controller/3.0.0 accepts/1.3.8 acorn/6.4.2 acorn/7.4.1 acorn/8.8.2 acorn-globals/4.3.4 acorn-jsx/5.3.2 #+end_example
** =floco show=
Print the =pdef= record for a given package.
#+BEGIN_SRC shell :exports both :results output nix run github:aakropotkin/floco -- show [email protected] --json; #+END_SRC
#+RESULTS: #+begin_example json { "ident": "lodash", "version": "4.17.21", "ltype": "file", "fetchInfo": { "narHash": "sha256-amyN064Yh6psvOfLgcpktd5dRNQStUYHHoIqiI6DMek=", "type": "tarball", "url": "https://registry.npmjs.org/lodash/-/lodash-4.17.21.tgz" }, "treeInfo": {} } #+end_example
- Core Scripts
=floco= uses a small collection of =bash= scripts to perform install tasks and drive builds.
These scripts do not depend on Nix, and are suitable for standalone use as replacements for ~pacote extract~ ( =install-module.sh= ) and ~(npm|yarn) run~ ( =run-script.sh= ).
You can install these as standalone executables using the =floco-utils= installable in the top-level flake.
For example usage and more details please see [[https://github.com/aakropotkin/floco/blob/main/doc/scripts/README.org#core-scripts][Core Scripts]].
- Updaters/Generators
The top level flake provides an installable =floco-updaters= as well as =app= targets ( =fromPlock= and =fromRegistry= ) that can be used to generate =pdefs.nix= and =pdefs.json= files to be loaded by as configs.
These scripts will allow you to convert existing JavaScript projects to be used with =floco=, and update/regenerate configs as projects' dependencies and build requirements change.
** =fromPlock=
This generator is intended for use with local projects. It is essentially a wrapper around =npm i --package-lock-only=.
For example usage please see the [[https://github.com/aakropotkin/floco/blob/main/doc/guides/basics.org][Getting Started]] guide.
** =fromRegistry=
This generator is intended for use with published registry packages that you'd like to make accessible to =floco= and =nix=.
This script behaves almost identically to =fromPlock=, except that it ignores =devDependencies= entirely, and accepts package descriptors as an argument ( as =npm= or =yarn= would ).
For example usage please see the [[https://github.com/aakropotkin/floco/blob/main/doc/guides/native-deps.org#preparing-a-workspace][Native Dependencies]] guide.
This script most useful for packaging executables and generating =treeInfo= information for packages that have =install= scripts ( such as =node-gyp= compilation ).
- Modules
Package metadata collection, also called /translation/, and project composition is managed using [[https://github.com/NixOS/nixpkgs/blob/master/lib/modules.nix][Nixpkgs Modules]] similar to those used by [[https://nixos.org/manual/nixos/stable/#sec-writing-modules][NixOS]], [[https://github.com/nix-community/dream2nix][dream2nix]], or [[https://github.com/nix-community/home-manager][home-manager]].
These modules are organized as sets of =interface.nix= and =implementation.nix= files and are designed to be extensible.
** Organization
The core of the module system revolves around a record called =pdef=, short for "package definition", which organizes translated metadata, and =package= records which organize the build pipelines.
This separation simplifies the organization of the /translation/ and /builder/ APIs, but the rationale runs further. The split allows us to flatly state: build routines must never perform impure operations, and translation routines must only produce fields that can be serialized to JSON.
Serialization of translated metadata allows Nix's =flake= features to drastically improve performance by leveraging [[https://www.tweag.io/blog/2020-06-25-eval-cache/][eval caching]] to avoid re-evaluation of recipe generation on successive runs.
** =pdef= Package Definitions
The =pdef= record closely mirrors the pseudo-standard schema used by most =package.json= files; but is much stricter about how declarations are written.
If desired, users could ditch =package.json= files altogether and simply write =pdef= records for their projects.
** Translators
At time of writing only a few translators have been migrated from the alpha iteration, [[https://github.com/aameen-tulip/at-node-nix][at-node-nix]], but in the near future these will be finalized for production use.
*** =package.json=
This is our bread and butter, and serves as the default implementation for creating a =pdef= record.
On its own this translator would require users to explicitly declare the structure of their =node_modules/= tree using the =treeInfo= submodule. For this reason we strongly recommend using the =package-lock.json= translator for projects with large dependency graphs.
**** Progress on /Ideal Tree/
The term /ideal tree/ refers to the mapping of a =node_modules/= tree from a dependency graph. This process is by far the most complex and challenging aspect of Node.js package management.
While =floco= currently relies on =npm= to generate /ideal trees/, this is expected to end soon.
The alpha repository [[https://github.com/aameen-tulip/at-node-nix][at-node-nix]] contains a large body of routines to perform /best effort/ =treeInfo= mapping, specifically handling projects which only require a single version of any package ( this property is called /The Golden Rule/ in package management contexts ).
Additionally the [[https://github.com/aameen-tulip/at-node-nix/blob/main/lib/sat.nix#L372][semver resolution]] routines used to fetch closures of /packument/ records effectively solve half of the /ideal tree/ process, leaving only scope and /follows/ management to be completed.
*** =package-lock.json= v2/v3
This is by far the most developed translator, and is the recommended option for large projects.
This translator will automatically fill =treeInfo= submodules, and triggers minimal network fetching.
*** =yarn.lock= v5
A rudimentary translator exists to collect information from =yarn.lock= v5 ( produced by =yarn= v3 ), but because these lockfiles lack /ideal tree/ information users will need to provide =treeInfo= themselves.
In the future we intend to produce =treeInfo= from these locks using the pinned version information they contain; but this routine still needs to be authored.
- Experimental Features
** =treeFor=
A CLI frontend for the =npm= /ideal tree/ routine, [[https://github.com/npm/cli/blob/main/workspaces/arborist/README.md][arborist]], modified such that =package-lock.json= files can be emitted to =STDOUT= without modifying the project.
This is expected to be used in later iterations of the /updaters/ allowing them to be run on ~/nix/store/~ paths. The =builtins.npmLock= example in the section takes advantage of this.
This executable is exposed as an installable and =app= in the top-level flake.
** Nix Plugin
A =nix= plugin for use with ~nix --plugin-files ...~ is available in the top level flake, along with a wrapper executable, =floco-nix=, which automatically loads this plugin.
In the future this plugin is expected to grow into a full executable that provides a suite of CLI commands; but for now it accepts =nix= arguments and sub-commands.
This plugin was developed for Nix v2.12.0, but is likely compatible with a wider range of versions.
*** New Builtins
Our plugin adds a few new =builtins= to the =nix= evaluator which are useful for dynamically generating package definitions.
**** =builtins.npmShow=
Wraps ~npm show~ allowing Nix to query package registries using a users existing =npm= config and any environment =NPM_CONFIG_*= variables.
While =floco= is already able to fetch package registry information without any external tools; this builtin is useful for accessing private package registries and inheriting authorization settings with minimal setup.
#+BEGIN_SRC shell :results output :exports both nix run github:aakropotkin/floco#floco-nix -- eval --json --expr ' builtins.attrNames ( builtins.npmShow "lodash" ) '|jq; #+END_SRC
#+RESULTS: #+begin_example json [ "_cached", "_contentLength", "_hasShrinkwrap", "_id", "_nodeVersion", "_npmOperationalInternal", "_npmUser", "_npmVersion", "_rev", "author", "bugs", "contributors", "description", "directories", "dist", "dist-tags", "gitHead", "homepage", "icon", "keywords", "license", "main", "maintainers", "name", "readmeFilename", "repository", "scripts", "time", "users", "version", "versions" ] #+end_example
**** =builtins.npmResolve=
Resolves package descriptors such as =foo@^1.0.0= or =bar@latest= using =npm=, returning a resolved URI.
This has the same environment and configuration properties as =npmShow=.
NOTE: if you use ranges such as [email protected]= you will want to use =builtins.split= to parse the output.
#+BEGIN_SRC shell :results output :exports both nix run github:aakropotkin/floco#floco-nix -- eval --expr ' builtins.npmResolve "lodash@latest" '; #+END_SRC
#+RESULTS: : "https://registry.npmjs.org/lodash/-/lodash-4.17.21.tgz"
**** =builtins.npmLock=
Produces a virtual =package-lock.json= for a given project path without modifying the project or making any writes to the filesystem.
This is an ideal alternative to the =fromRegistry= /updater/ when used in combination with =builtins.fetchTree= and =builtins.npmResolve=.
In practice you can dynamically generate full dependency closures' =treeInfo= records using this routine. I currently use it for this purpose out in the field; but have avoided using it in the default modules so that they are usable without plugins.
#+BEGIN_SRC shell :results output :exports both
nix run github:aakropotkin/floco#floco-nix -- eval --impure
--expr 'let
url = builtins.npmResolve "pacote@latest";
src = builtins.fetchTree { type = "tarball"; inherit url; };
plock = builtins.npmLock src;
in builtins.attrNames plock
';
#+END_SRC
#+RESULTS: : [ "lockfileVersion" "name" "packages" "requires" "version" ]
**** =builtins.semverSat=
Runs =node-semver= to test whether a semantic version satisfies a constraint. In the future =node-semver= will be replaced using a native C++ port [[https://github.com/aakropotkin/semi.git][semi]].
This largely exists as a stop-gap until the pure =nix= implementation from the alpha repository is polished and/or =semi= is completed.
#+BEGIN_SRC shell :results output :exports both nix run github:aakropotkin/floco#floco-nix -- eval --expr '[ ( builtins.semverSat "^4.2.0" "4.0.0" ) ( builtins.semverSat "^4.2.0" "4.2.0" ) ( builtins.semverSat "^4.2.0" "4.2.1" ) ( builtins.semverSat "^4.2.0" "4.3.0" ) ] '; #+END_SRC
#+RESULTS: : [ false true true true ]
- Future Extensions
Many of the following extensions have function drafts or well tested prototypes in the alpha release of =floco=; but are not developed enough for use in production code-bases as pieces of reliable infrastructure.
- Improved support for package.json workspaces.
- Currently reliance on =npm= and special configuration based on in depth knowledge of =floco= is necessary to accomplish workspace support.
- Practically a template or example using workspaces is likely sufficient for the immediate future; but the NixOS Module system is expected to resolve issues that previously made workspaces complex to manage.
- Expanded CLI tooling.
- Currently users are asked to interact with nix to drive builds, tests, update metadata, etc. Ideally a simple bash script would provide familiar commands such as ~floco add <PKG>~, ~floco publish~, ~floco update~, ~floco build~, etc that =npm= and =yarn= users are already familiar with.
- Nix plugin to read/write caches globally and into =flake.lock=.
- This is the real end goal for =floco=. It should be possible to read/write =floco= metadata to =flake.lock= and existing =nix= caches.
- There is currently a draft plugin which allows nix to adopt =npm= URIs to refer to packages as [email protected]= which could be expanded upon.
- Project templates and propagation of build recipes could allow =nix= to abstract away the generation of =flake.nix= for the vast majority of projects which would be a significant UX breakthrough.
- Semantic version parsing, and /ideal tree/ formation.
- Currently =floco= really relies on =npm= and its =package-lock.json= to construct non-trivial node_module/ metadata declarations. This reliance is a major pain point for handling projects which currently use yarn since interoperability between =yarn= and =npm= across their associated lockfiles is implemented incredibly poorly, to such a degree that you cannot trust them to behave predictably in the same source tree.
- Semver parsing and solving SAT is implemented in the alpha repository, and
has been testing on large non-trivial inputs quite successfully.
Still this effort requires a few weeks of polishing to really approve for
use in production.
- For now we have provided [[https://github.com/npm/node-semver.git][node-semver]] as an installable in the top-level flake for use in scripts and our [[https://github.com/aakropotkin/floco/blob/main/pkgs/nix-plugin][floco-nix]] through =builtins.semverSat=.
- Construction of ideal tree from semver SAT is a project in and of itself in order to support things like =optionDependencies=, =peerDependencies=, =bundledDependencies=, and other oddballs which are a prerequisite for use in the general case.
- Community
** Matrix Sadly IRC is dead. IRC remains dead. So like most folks these days we use Matrix Chat.
Space: [[https://matrix.to/#/#floco:matrix.org][#floco:matrix.org]]
General Room: [[https://matrix.to/#/!wMSeevIIjIbAOVbqHh:matrix.org?via=matrix.org]] ( Recommended )
Support Room: [[https://matrix.to/#/!tBPFHeGmZfhbuYgvcw:matrix.org?via=matrix.org]]
Development Room: [[https://matrix.to/#/!qDFpEnHkbpkhLSenko:matrix.org?via=matrix.org]]
- Supporters
** [[https://tulip.co/][Tulip Interfaces]] =floco= was originally developed for use by Tulip Interfaces. Without their support this project never would have been possible.
aakropotkin