welo icon indicating copy to clipboard operation
welo copied to clipboard

Iterative and Concurrent Traversal

Open tabcat opened this issue 2 years ago • 1 comments

The traverser needs to be turned into an async iterator. The reason this hasn't been done yet is mainly time and difficulty. The traverser needs to be able to traverse efficiently for ordered and unordered traversal so there is no duplicate code. Fetching and caching entries ahead of when they are needed would be nice.

IIRC the difficulty is doing this with the using the yield statement. May need to make a custom AsyncGenerator

  • [ ] Traverser Function returns Async-Iterator
  • [ ] Replica.traverse returns an Async-Iterator
  • [ ] Store API consumes Async-Iterator
  • [ ] IPFS/Pubsub Heads Exchange Replicator consumes Async-Iterator

tabcat avatar Feb 04 '23 02:02 tabcat

might be good to separate this part out into a utils package or its own as it's generally useful

tabcat avatar Aug 15 '23 16:08 tabcat