firewalk

firewalk copied to clipboard

I think that something exposing an async iterator would be cleaner to use.

  let count = 0;
  let queue = [];
  for await (const snapshot of createTraverser(...)) {
    count++;
    queue.push(sendEmail(snapshot.data());
    if (queue.length >= 50) {
      await promise.all(queue);
      queue = [];
    }
  }
  await promise.all(queue);
  console.log(`Processed ${count} users.`);

The items can simply implement the async iterator methods, and fetch in batches internally.

Note: I added the queue/batch stacking for the outbound requests... but for other use cases, would still be able to inline the await, a record at a time, such as dumping to a users.jsonl.gz file or similar.

@tracker1 Thanks for the suggestion and for creating the very first issue :) So it looks like we would need to create a new Traversal object that will implement async iterable. Traverser itself can't be iterable since it's an object that does the traversal. Also, it looks like we would need to sacrifice the TraversalResult object in that case, correct?

Let's keep this issue open and I'll definitely think more about this in the coming days. Also curious what other users have to say about this.