celestia-core
celestia-core copied to clipboard
Use file-based block store
Currently, the blockstore uses a key-value store (like leveldb or pebbledb) that are LSM-based. This isn't great for multiple reasons:
- Celestia blocks are very large, while these databases are usually optimized for smaller values.
- The overhead of LSM-based key-value stores isn't needed since blocks are append-only by construction with monotonic keys (block height), rather than having random keys.
- Pruning in these databases is involved, and is currently very unreliable (i.e. may actually not even happen at all).
Add an option to allow the blockstore to use a file-backed storage. Ideally for all data in the block, not just the block body, such that a key-value store for rest of the block data isn't needed (header, commit, etc.). Initially, the file format can just be something like a protobuf encoding of what currently goes as the value in the key-value store. As for the key (i.e. filename), it can be the height of the block.
One thing to watch for are filesystem limits on the number of files in a directory. If all blocks are stored in a single directory, perhaps this limit might become a problem (not sure, so please confirm). Other utilities can also be affected by large number of files in a single directory (e.g. >10,000).