silkworm icon indicating copy to clipboard operation
silkworm copied to clipboard

Very first sync from scratch History index causes HUGE etl files - limit batch ?

Open AndreaLanfranchi opened this issue 2 years ago • 0 comments

During very first sync from scratch the execution of Stage History Index causes a lot of etl files to be flushed into etl directory. On ethereum mainnet (at block ~15.4M) I recorded up to 150GB of etl data which eventually translates into the same amount of space when loaded into db. This implies when the db is already large enough (before History Index db is 1.65TB) user needs at least 300GB extra space for the stage to complete. Eventually etl gets freed.

Worth to investigate if it's viable to run History Index in batches (pretty much like Execution) to limit the generation of huge temporary data and make it restartable.

AndreaLanfranchi avatar Sep 01 '22 10:09 AndreaLanfranchi