db: state snapshots

[canepat]

One of the main features of Erigon3 is moving most of the state content (i.e. account/storage/code) and its validity proof (i.e. commitment) to snapshot flat files. This is accomplished by introducing new abstractions for modelling the common requirements these entities have in terms storage, starting from the kind of queries they must support.

Obviously, some refactoring of the existing snapshot support is required in order to accommodate the new abstractions in the cleanest, most coherent and ultimately most useful way.

This is the main tracking issue. Individual issues are listed below:

• [x] b+tree index implementation #2367
• [x] bloom filter and existence index #2386
• [x] layering: separation between dal and datastore in db module #2416 #2424 #2427
• [x] decouple SnapshotType from SnapshotPath and generalisation of SnapshotPath #2419
• [x] definition of timed-series unit (e.g. Timestamp) and move to dal package #2416 #2419
• [x] refactoring of Snapshot, SnapshotReader as Segment, SegmentReader #2424
• [x] introduce KVSegment: 3 types of iterators - pairs, keys, values #2509 #2518 #2526
• [x] schema definition
  • [x] entity identifier type #2467 #2491
  • [x] decouple schema (i.e. SnapshotBundleData) from bundle and move to dal package #2484
  • [x] use fluent config (a.k.a. builder) pattern for DDL-like definition of entities #2484
• [ ] index support evolution
  • [x] use specific type directly as index (e.g. RecSplitIndex) and remove Index class
  • [ ] refactor IndexBuilder: multiple methods for each type or multiple builders
• [ ] segment/index decoders
  • [x] segment decoders #2544
  • [x] inverted index decoders #2544
  • [ ] index reader: new abstraction like Segment which decouples K/I serialization
• [x] snapshot repository
  • [x] refactor to work with any timestamp (BlockNum, TxStep) #2437
  • [x] refactor to support multiple snapshot repository instances #2440
  • [x] 2 instances as starting point: block snapshots and state snapshots #2467 #2491
• [x] state snapshot repository #2494 #2509 #2538
• [x] state snapshot path #2507
• [ ] new KVDB data model (i.e. MDBX)
  • [x] refactor datastore in preparation to add D/H/II #2559
  • [x] D/H/II schema definition #2582
  • [ ] implement II encoder/decoder in KVDB #2614 #2632
  • [ ] implement H encoder/decoder in KVDB #2614
  • [ ] implement D encoder/decoder in KVDB #2614
• [ ] new D/H/II queries
  • [x] analysis to figure out which queries are needed for D/H/II ("query objects" pattern)
  • [ ] implementation of queries in KVDB #2614 #2632
  • [ ] implementation of queries in snapshots #2625
  • [ ] helper to create queries (i.e. factory)
• [ ] Data API a.k.a. Temporal KV
  • [ ] implement Inverted Index API on top of queries
  • [ ] implement History API on top of queries
  • [ ] implement Domain API on top of queries
  • [ ] gRPC D/II Data API server
• [ ] Support state snapshots in C API and in silkworm-go