reth icon indicating copy to clipboard operation
reth copied to clipboard

feat(stages): log indexing

Open rkrasiuk opened this issue 2 years ago • 4 comments

Description

This PR introduces log address and topic indexing - supplementary indices for log querying (intended to be used by RPC module).

Two new tables are added to the database:

  1. LogAddressHistory - mapping of log address to block numbers where the logs were emitted by this address
  2. LogTopicHistory - mapping of log topic to block numbers where this topic occurred.

The IndexLogHistoryStage walks over Receipts table and creates indices for log addresses and topics in corresponding tables.

rkrasiuk avatar Jun 09 '23 13:06 rkrasiuk

Good start..

  1. Can we run this on an existing node to backfill via reth stage run --commit?
  2. How big are the tables on mainnet at tip?

gakonst avatar Jun 09 '23 17:06 gakonst

@gakonst https://github.com/paradigmxyz/reth/pull/3089 solves this by starting the pipeline for any newly added stage

rkrasiuk avatar Jun 12 '23 11:06 rkrasiuk

Codecov Report

Merging #3082 (c4131ca) into main (428a6dc) will decrease coverage by 0.09%. The diff coverage is 48.81%.

Impacted file tree graph

Impacted Files Coverage Δ
bin/reth/src/args/rpc_server_args.rs 47.22% <0.00%> (-0.40%) :arrow_down:
bin/reth/src/args/stage_args.rs 0.00% <ø> (ø)
bin/reth/src/stage/drop.rs 2.00% <0.00%> (-0.13%) :arrow_down:
crates/consensus/common/src/validation.rs 78.68% <0.00%> (-0.43%) :arrow_down:
crates/interfaces/src/test_utils/generators.rs 98.15% <ø> (+7.36%) :arrow_up:
crates/primitives/src/lib.rs 100.00% <ø> (ø)
crates/primitives/src/log.rs 100.00% <ø> (ø)
crates/rpc/rpc-builder/src/auth.rs 44.16% <0.00%> (-0.46%) :arrow_down:
crates/rpc/rpc-builder/src/lib.rs 67.49% <0.00%> (-0.15%) :arrow_down:
crates/stages/src/sets.rs 0.00% <0.00%> (ø)
... and 20 more

... and 7 files with indirect coverage changes

Flag Coverage Δ
integration-tests 15.96% <0.00%> (-0.13%) :arrow_down:
unit-tests 63.89% <48.81%> (-0.06%) :arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
reth binary 26.36% <0.00%> (-0.06%) :arrow_down:
blockchain tree 81.25% <ø> (ø)
pipeline 87.00% <90.11%> (+0.09%) :arrow_up:
storage (db) 72.50% <34.38%> (-0.99%) :arrow_down:
trie 94.66% <ø> (ø)
txpool 49.11% <ø> (ø)
networking 77.91% <ø> (+0.02%) :arrow_up:
rpc 57.69% <6.14%> (-0.36%) :arrow_down:
consensus 62.04% <0.00%> (-0.07%) :arrow_down:
revm 34.90% <ø> (ø)
payload builder 6.83% <ø> (ø)
primitives 88.54% <100.00%> (+0.26%) :arrow_up:

codecov[bot] avatar Jun 12 '23 12:06 codecov[bot]

There's a conflict here now, sorry about that. Did we sync a mainnet node with this to sync how big the DB grew by it?

gakonst avatar Jun 13 '23 07:06 gakonst

@gakonst We need to figure out when to release this - there is another breaking DB change in #3464, so I wonder if we should bundle both in e.g. Alpha 3?

onbjerg avatar Jun 29 '23 20:06 onbjerg