silkworm
silkworm copied to clipboard
Very first sync from scratch History index causes HUGE etl files - limit batch ?
During very first sync from scratch the execution of Stage History Index causes a lot of etl files to be flushed into etl
directory.
On ethereum mainnet (at block ~15.4M) I recorded up to 150GB of etl data which eventually translates into the same amount of space when loaded into db. This implies when the db is already large enough (before History Index db is 1.65TB) user needs at least 300GB extra space for the stage to complete. Eventually etl gets freed.
Worth to investigate if it's viable to run History Index in batches (pretty much like Execution) to limit the generation of huge temporary data and make it restartable.