Store headers in memory efficiently

[kostyantyn]

Is your feature request related to a problem? Please describe Bitcoin creates a block every 10 minutes or ~51 840/year. As one header (CBlockIndex) weights ~ 204bytes (rough calculations) it's only 10MB/year. It means they can easily keep all the headers in memory.

In our case, if we go with 16sec/block, we will have ~1 944 000 headers/year or ~378MB/year. With this scale, we would need to come up with an alternative solution on how to keep them in memory efficiently.

Describe the solution you'd like Technically we can keep only headers from the last finalization we have. All other headers are not needed, as we don't re-org before the finalization. However, we would need to provide such headers when new nodes join the network. In this case, we can look up the disk.

Also, to reduce the disk size, we can trim accumulated fields, e.g.: stake_modifier or chain_stake (that we plan to add). Currently we store/plan to store them in the every CBlockIndex. We already have in place the mechanism that deletes old finalization transactions from the CBlockIndex. We can apply it to delete other fields or even create a separate data structure for fields which need to be cleaned up.

Describe alternatives you've considered Haven't reviewed alternatives yet.

[frolosofsky]

We also started to store commits in the CBlockIndex, and it increases its size significantly.

[thothd]

regardless of the specific data we decide to store and how efficient it's gonna be. with frequent blocks (e.g 16 seconds) and the need for new clients to validate the chain when syncing. we gonna run in challenges of storing headers (+ extra data) in memory. we should analyse the entire disk/memory use cases and come with a solution for paging data in/out dynamically per node depending on finalization progress of the current tip and whether clients are syncing with the node