feat: add StaticFileSegment::AccountChangeSets
ref https://github.com/paradigmxyz/reth/issues/18846
Introduces a new flag --db.enable-v2-static-files which controls whether or not we should write account changesets to static files or the DB.
Adds account changesets as a new static file segment, where each row is a change. This makes them different from "block-based" static files like headers, and "transaction-based" static files like transactions and receipts. Row ranges for each block are stored in the header for the static file segment.
Backwards compat deserialization code is added because this adds fields to the static file header.
Super excited for this
CodSpeed Performance Report
Merging #18882 will not alter performance
Comparing dan/account-changeset-static-files (c79604f) with main (cc7edeb)
Summary
✅ 77 untouched
⏩ 4 skipped[^skipped]
[^skipped]: 4 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.
we might want to try this one with lz4
https://github.com/paradigmxyz/reth/blob/f0c0b3db4e47c167244654a5f03e5e388c9505fe/crates/storage/provider/src/providers/static_file/writer.rs#L833-L837
figured out why this is causing state root mismatches sometimes, we are still using raw txs to access changesets in AccountExtReader:
https://github.com/paradigmxyz/reth/blob/f2193a1b64be4bb80a3dbe9654a93acb9dda7273/crates/storage/provider/src/providers/database/provider.rs#L854
https://github.com/paradigmxyz/reth/blob/f2193a1b64be4bb80a3dbe9654a93acb9dda7273/crates/storage/provider/src/providers/database/provider.rs#L831
this causes problems for account hashing, particularly when you have already run the pipeline once
Update, this no longer has state root mismatches. Testing running this on a very large range of blocks to compare size versus the table in mdbx.
TODOs required for this PR to be complete so far:
- [x] Introduce method
is_changeset_basedinstead ofis_block_based/is_transaction_based - [x] Use
StorageSettingsto determine behavior when reading / writing - [x] Look into
test_segment_config_backwards - [x] Remove traces added
- [x] Put
increment_blockinappend_account_changeset - [x] Fix IndexAccountHistory collector to walk incrementally rather than collect all changesets in range