flow-go icon indicating copy to clipboard operation
flow-go copied to clipboard

Published 20 hours ago • verified publisher icon onflow

[EN Performance] Determine if checkpoint file should be split up and processed in parallel to resolve several issues

Open fxamacker opened this issue 3 years ago • 0 comments

Problem

Checkpoint file is around 160GB and growing. Also, state extraction during mainnet spork should be faster. Lastly, we currently evict entire 160GB checkpoint file from Linux page cache when the file is closed.

Updates #1744

The Proposed Solution

Determine if we should split up the checkpoint file and process portions in parallel to:

  • Speedup checkpointing.
  • Speedup state extraction during mainnet spork.
  • Eliminate the need to evict segments of 160GB checkpoint file from Linux page cache during use.

PR #3050 (merged on Aug 23 and not deployed to August Spork) set the foundation for parallel processing of subtries by splitting up the mtrie into 16 subtries. It will reduce peak RAM use (max RSS) by dozens of GB once deployed. Parallel processing in future PR will use up of some or all of that memory reduction.

Caveats

Considerations include third parties who currently read and parse the checkpoint file with their own software.

fxamacker avatar Aug 23 '22 22:08 fxamacker