celestia-core icon indicating copy to clipboard operation
celestia-core copied to clipboard

Use file-based block store

Open adlerjohn opened this issue 5 months ago • 3 comments

Currently, the blockstore uses a key-value store (like leveldb or pebbledb) that are LSM-based. This isn't great for multiple reasons:

  1. Celestia blocks are very large, while these databases are usually optimized for smaller values.
  2. The overhead of LSM-based key-value stores isn't needed since blocks are append-only by construction with monotonic keys (block height), rather than having random keys.
  3. Pruning in these databases is involved, and is currently very unreliable (i.e. may actually not even happen at all).

Add an option to allow the blockstore to use a file-backed storage. Ideally for all data in the block, not just the block body, such that a key-value store for rest of the block data isn't needed (header, commit, etc.). Initially, the file format can just be something like a protobuf encoding of what currently goes as the value in the key-value store. As for the key (i.e. filename), it can be the height of the block.

One thing to watch for are filesystem limits on the number of files in a directory. If all blocks are stored in a single directory, perhaps this limit might become a problem (not sure, so please confirm). Other utilities can also be affected by large number of files in a single directory (e.g. >10,000).

adlerjohn avatar May 05 '25 14:05 adlerjohn