reth icon indicating copy to clipboard operation
reth copied to clipboard

feat: add StaticFileSegment::AccountChangeSets

Open Rjected opened this issue 2 months ago • 5 comments

ref https://github.com/paradigmxyz/reth/issues/18846

Introduces a new flag --db.enable-v2-static-files which controls whether or not we should write account changesets to static files or the DB.

Adds account changesets as a new static file segment, where each row is a change. This makes them different from "block-based" static files like headers, and "transaction-based" static files like transactions and receipts. Row ranges for each block are stored in the header for the static file segment.

Backwards compat deserialization code is added because this adds fields to the static file header.

Rjected avatar Oct 06 '25 18:10 Rjected

Super excited for this

gakonst avatar Oct 06 '25 21:10 gakonst

CodSpeed Performance Report

Merging #18882 will not alter performance

Comparing dan/account-changeset-static-files (c79604f) with main (cc7edeb)

Summary

✅ 77 untouched
⏩ 4 skipped[^skipped]

[^skipped]: 4 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

codspeed-hq[bot] avatar Oct 07 '25 16:10 codspeed-hq[bot]

we might want to try this one with lz4

https://github.com/paradigmxyz/reth/blob/f0c0b3db4e47c167244654a5f03e5e388c9505fe/crates/storage/provider/src/providers/static_file/writer.rs#L833-L837

joshieDo avatar Oct 21 '25 10:10 joshieDo

figured out why this is causing state root mismatches sometimes, we are still using raw txs to access changesets in AccountExtReader: https://github.com/paradigmxyz/reth/blob/f2193a1b64be4bb80a3dbe9654a93acb9dda7273/crates/storage/provider/src/providers/database/provider.rs#L854

https://github.com/paradigmxyz/reth/blob/f2193a1b64be4bb80a3dbe9654a93acb9dda7273/crates/storage/provider/src/providers/database/provider.rs#L831

this causes problems for account hashing, particularly when you have already run the pipeline once

Rjected avatar Nov 03 '25 21:11 Rjected

Update, this no longer has state root mismatches. Testing running this on a very large range of blocks to compare size versus the table in mdbx.

TODOs required for this PR to be complete so far:

  • [x] Introduce method is_changeset_based instead of is_block_based / is_transaction_based
  • [x] Use StorageSettings to determine behavior when reading / writing
  • [x] Look into test_segment_config_backwards
  • [x] Remove traces added
  • [x] Put increment_block in append_account_changeset
  • [x] Fix IndexAccountHistory collector to walk incrementally rather than collect all changesets in range

Rjected avatar Nov 10 '25 21:11 Rjected