Epic: Portable, Auto-detected IPFS Datasets

[flyingzumwalt]

This is a catch-all for the functionality around ipfs-pack, which is described in the ipfs-pack draft proposal

High-level Value Proposition

For the first pass, ipfs-pack will help us support the use case where users Use Manifest Files to Track Directory Structure & Contents, which allows us to Track a Directory and Serve it on IPFS without making duplicate local copies of the data. This will eventually allow us to Round-trip whole directories through IPFS and Mount directories by auto-detecting their ipfs-pack manifests or prebuilt object databases

The case for ipfs-pack

Currently the way people use go-ipfs is with ipfs add which creates a duplicate copy of the added data on the machine. With filestore we aim to build indexes of pointers to data/blocks in-place. This solves performance concerns, but creates a brittle situation -- if you move the file, ipfs won't be able to serve it any more. ipfs-pack aims to address this by building manifest files that hold the indexes that match ipfs hashes to the content. If you store those manifest files alongside the cotnent they point to, it becomes a portable dataset.

Extending that idea, if you create little .ipfs repositories next to the manifest files, it becomes possible to

• serve that dataset as its own little ipfs node
• register the contents of that dataset with another ipfs node, serving the content directly from wherever you've stored/mounted it