lighthouse
lighthouse copied to clipboard
Compress and linearise beacon blocks on disk
Description
Beacon blocks alone now consume around 80GB of disk space on mainnet.
I think we could be more efficient with storage by using compression. Recently I'm particularly fond of zstd compression, which is fast and could probably offer a 2-4x reduction.
The complicating factor is that compressing blocks requires a database schema upgrade, as well as a moderately sophisticated migration to copy and compress all the existing blocks.
Another change we could implement at the same time would be linearising finalized blocks in the freezer database. Currently we key them by block root indefinitely, which leads to lots of non-sequential reads. Finalized blocks could be keyed by slot, which would make it quicker to replay them or serve them via BlocksByRange.