go-car icon indicating copy to clipboard operation
go-car copied to clipboard

carv2/ReadWriteBlockstore: support deferred root CIDs

Open raulk opened this issue 2 years ago • 5 comments

blockstore.OpenReadWrite requires providing the root CIDs when creating a new blockstore. This design inhibits the ability to use a CARv2 blockstore as the target of a streaming merkle DAG formation like UnixFS, as the root CID is not known beforehand.

We could work around this situation by supplying a placeholder root CID, and once the blockstore is finalized, we could go back and replace those bytes in the header.

Unfortunately, the library doesn't provide APIs to do that safely and without making assumptions about the underlying format, or breaking abstractions.

Some ideas:

  1. Allow blockstore.OpenReadWrite to take a []struct{cid.Builder, func() (cid.Cid, error)}.
    • Use the cid.Builders to compute the length of each CID, and preallocate those bytes in the header.
    • On Finalize, call each function to get the actual CID to replace it in the header.
  2. Simpler: allow the user to specify a number of bytes to preallocate, and provide primitives to update a CAR header.

raulk avatar Aug 02 '21 20:08 raulk