lighthouse icon indicating copy to clipboard operation
lighthouse copied to clipboard

Compress and linearise beacon blocks on disk

Open michaelsproul opened this issue 3 years ago • 1 comments

Description

Beacon blocks alone now consume around 80GB of disk space on mainnet.

I think we could be more efficient with storage by using compression. Recently I'm particularly fond of zstd compression, which is fast and could probably offer a 2-4x reduction.

The complicating factor is that compressing blocks requires a database schema upgrade, as well as a moderately sophisticated migration to copy and compress all the existing blocks.

michaelsproul avatar May 24 '22 00:05 michaelsproul

Another change we could implement at the same time would be linearising finalized blocks in the freezer database. Currently we key them by block root indefinitely, which leads to lots of non-sequential reads. Finalized blocks could be keyed by slot, which would make it quicker to replay them or serve them via BlocksByRange.

michaelsproul avatar May 24 '22 00:05 michaelsproul