pkg/store: init

[flokli]

This provides a bunch of store-related components and concepts. Very WIP. PR opened for some early feedback.

• A "Path Store", storing information about an output path (some .narinfo-style metadata, and the listing of .nar files)

• A "Chunk Store", storing content-adressed chunks

• Something converting from .narinfo and .nar to that format, plus tests.

• I'm not yet sure about the package layout. pkg/nixpath feels a bit alien, and I'd probably like to have two packages, one for "path metadata and stores", and one for chunks (?)

• There still needs to be a store exposing a "legacy" binary cache with these new interfaces (populating a "Path Store" and "Chunk Store" on access).

• There still needs to be something converting back from these two stores to .nar and .narinfo

• Once that's done, we probably want to write an HTTP handler, so nix copy to it, and nix-store -r from it becomes possible.

• There needs to be more tests (for all chunk stores, for all path stores).

[flokli]

I was surprised that the SimpleChunker produced only one chunk. For some reason, I assumed it would output fixed-sized chunks (modulo the last one).

It is explained in the description though ;-) It's picking up one of the ideas from https://alternativebit.fr/posts/nixos/future-of-nix-substitution/, which hashes each file inside the .nar individually. It might make sense to not involve a complicated chunker if the file is not much bigger than a defined file size, so this might entirely get absorbed into the import logic. If not, we could probably give it a better name at least :-)

How useful is it to keep the offset with the chunk? Potentially it could be returned by the Next() function. Or if each Chunk has a Size(), then a list of chunks is enough to re-compose the original file.

It wasn't really useful - adressed in https://github.com/nix-community/go-nix/pull/86/commits/ecf6906bb84ba74ffda0fdf38f60b4af9687d931. Each chunk now is just an alias to []byte.