silkworm icon indicating copy to clipboard operation
silkworm copied to clipboard

Db format for Account/Storage History indexes

Open AndreaLanfranchi opened this issue 2 years ago • 1 comments

Current implementation (compatible with erigon stable) defines the indexes schema as :

k : key + bitmap_shard_upper_limit (BE uint64_t)
v : bitmap data

where key is :

  • for AccountHistory the account address
  • for StorageHistory the composite of contract address + location

Need to evaluate if is worth to change the table definitions to dupsorted (aka multivalue) as :

k : key
v : bitmap_shard_upper_limit (BE uint64_t) + bitmap data

There should be minimal changes in the reads of historical accounts/storages and lot less key duplications

AndreaLanfranchi avatar Aug 25 '22 07:08 AndreaLanfranchi

As @AskAlexSharov points out in multi-value mode the size of v is constrained to keySizeLimit (I honestly didn't think about that) Need to stat how many shards per account/location and average shard data size to find a sweet spot.

AndreaLanfranchi avatar Aug 25 '22 07:08 AndreaLanfranchi