Indexes don't take advantage of performant structure

[willscott]

The common form of Index introduced in Car V2 is to lay out the CIDs in the car in a sorted table that can be quickly walked through with a binary search.

We should attempt to optimize not fully copying the index into memory when loading a car, and just doing spot reads from the index. This can for large cars avoid a copy of hundreds of megs per individual block read.

[mvdan]

Dupe of https://github.com/ipld/go-car/issues/243 or https://github.com/ipld/go-car/issues/95, perhaps. We already have multiple issues that all boil down to: we should be able to consume some index formats directly from disk.

[willscott]

yep, happy to dupe, maybe this is more trying to indicate that this remains one of our expected performance bottle necks that we shouldn't lose track of

[aarshkshah1992]

Would be GREAT to have for https://github.com/filecoin-project/dagstore/pull/116.