Chunk store implementation

[aloknerurkar]

Chunk store

Implements the ChunkStore abstraction defined in the storage abstractions. The is the base storage layer for the Chunks and will be used by all the components to be implemented as part of the Storage rewrite.

In the current implementation, we have sharky, which is a very optimal way to store fixed-length blobs on disk. The ChunkStore will basically be a combination of this along with the retrievalIndex which maintains the mapping of swarm.Address to the sharky location. The ChunkStore will also store stamp information. For future proofing, we need to separate the stamp information from the retrieval Index. This would allow one-to-many mapping between chunks and stamps.

type chunkIndex struct {
        Addr swarm.Address
        RefCnt uint8
}

type chunkBatch struct {
        Addr swarm.Address
        BatchId []byte
        BatchIndex uint64
        Sig []byte
}

The ChunkStore will also need to do reference counting. As multiple components will use the Chunk store, a chunk could possibly be shared among components. In order to prevent any duplicate storage of chunks, we will use a simple reference counting scheme. Once the refCnt drops to 0, the chunk can be released from sharky.

Things to consider

• Metrics around get/put/delete operations. Also, metrics related to sharky.