pathfinder icon indicating copy to clipboard operation
pathfinder copied to clipboard

WIP: feat(pathfinder/sync/l2): add a bulk fetch mode to feeder gateway sync

Open kkovaacs opened this issue 6 months ago • 0 comments

Just like we do checkpoint sync as a first step of a P2P sync this change implements a special "bulk" sync mode for feeder gateway sync.

In bulk mode we do not expect reorgs to happen so the fetch logic is way simpler. We also concurrently fetch and verify data for multiple blocks at once, improving fetching throughput even if latency is high.

Block fetching and verification now happens in individual futures of a buffered stream. The level of buffering determines the level of concurrency we're doing with fetching. To avoid deadlocks all events are emitted in the consumer of the stream.

Initial measurements are very promising: together with the Pedersen hash improvements we can now sync the first 80k blocks of the Sepolia testnet in ~1h 30s.

TODO:

  • [ ] add tests for bulk sync
  • [ ] add CLI option for setting download concurrency level

kkovaacs avatar Aug 19 '24 14:08 kkovaacs