NixIPFS
Build Input Caching
Beside distributing binary files, the build inputs of a derivation could be cached in IPFS. That would speed up CI quite a bit since the cache could move "closer" to the build instance (using an IPFS Gateway that is within the local network).
Current Status: @knupfer created fetchIPFS: https://github.com/NixOS/nix/issues/859#issuecomment-241787645 @CMCDragonkai is doing something similiar @ https://github.com/MatrixAI/Forge-Package-Archiving NAR replacement discussion: https://github.com/NixOS/nix/issues/1006
There are different methods on how to approach this:
- put build inputs into IPFS and make a
sha256:IPFSmapping inside nixpkgs (a huge set) Migration: easy (can be solved by a script that builds all /nix/store/ paths and puts them into IPFS) Support: every fetch derivation needs to be patched to look up the IPFS hash first, either fetch it from the local running daemon or a gateway. If no hash is found each derivation falls back to its normal operation (curl, git, svn etc.) - append the IPFS hash to each package (@knupfer's idea). Migration: takes a lot of effort (each derivation must be touched) Support: already implemented, however it must be added manually to each package
Should we produce an intermediate cache format (tar? -> can be chunked efficiently in IPFS) that is used to cache each src? This can then be used within each fetch derivation. As a big plus, the build inputs are deduplicated and share blocks with the official package.
However it would be better just to reuse an already archived src (like tar.*, .zip etc.) and not to archive it again -> We need some form of manifest -> We need IPLD
We are currently in the midst of going through ipfs internals (specifically libp2p) in order to create a haskell implementation and have a deeper integration of build input source hashes and ipfs architecture.
The biggest benefit is the ability to version srcs that are currently not versionable. Loads of packages distributors don't keep historical versions.
We're halfway through implementing the haskell multiaddress library: https://github.com/MatrixAI/haskell-multiaddr (see improved branch). Still to do - binary encoding/decoding.
Perhaps we should meetup and compare notes?
This script prints all tarballs for an expression:
nix-instantiate --eval --json --strict maintainers/scripts/find-tarballs.nix --arg expr '(import <nixpkgs> {}).hello'
there is also maintainers/scripts/copy-tarballs.pl that uses it to upload to S3
First implementation: https://github.com/NixIPFS/nixipfs-scripts/blob/master/nixipfs/src/mirror_tarballs.py