fastcrawler
fastcrawler copied to clipboard
Define batching impl
Decouple batching and performing the request, by using a new class called Batcher
def get_batches(self)-> iterable[Batch]:
return list(Batch(engine=self.engine, setting=self.setting, ...))
for batch in self.get_barches():
batch_results = await Batch.perform()
parsed_result = Parse(batch_result)