lotus icon indicating copy to clipboard operation
lotus copied to clipboard

Enable efficient indexing of historical chain data

Open fridrik01 opened this issue 1 year ago • 4 comments

See https://github.com/filecoin-project/fvm-pm/issues/299

Lotus currently uses the following sqlite databases:

  • sqlite/events.db: Stores events sent by actors in the FVM
  • sqlite/txhash.db: Stores mappings of Eth tx hash to Filecoin message cid
  • sqlite/msgindex.db: Stores block messages cid and their tipset cid for faster lookup

We should try to unify all the different databases into one which should make maintaining correctness and doing recovery/backfilling simpler.

Also, we make sure we handle the following:

  • Make sure we handle all edge cases (forks, reverts, config changes that require pruning, etc)
  • Allow enabling/disabling what to index
  • ~~Be able to configure lookback so these indices don't grow endlessly~~ Not needed as the indexes are tiny compared to chain data (less than 0.25%)
  • #11007

fridrik01 avatar May 03 '23 12:05 fridrik01