Support Data Segment Index

[willscott]

Boost component

• [X] boost daemon - storage providers

Improvement Suggestion

Currently boost indexes pieces using go-car to generate a local index of CIDs contained in the piece for use in retrievals and to notify network indexers.

This format has been extended recently, per the go-data-segment library. This wrapper can be used to parse pieces, in particular by parsing a trailing index if it exists.

as @kubuxu notes:

Here is the code for parsing the data segment index: https://github.com/filecoin-project/go-data-segment/blob/master/datasegment/parse_index.go After the index is parsed, you should use IndexData#ValidEntries()` to gather valid data segments on which you can access information of each segment using PieceCID() UnpaddedOffset() UnpaddedLength(): https://github.com/filecoin-project/go-data-segment/blob/master/datasegment/index.go#L156-L173

This is currently managed by the dagstore I think the lid variant will be this? (can you confirm, @nonsense?)