fastcrawler icon indicating copy to clipboard operation
fastcrawler copied to clipboard

Define batching impl

Open ManiMozaffar opened this issue 2 years ago • 0 comments

Decouple batching and performing the request, by using a new class called Batcher

def get_batches(self)-> iterable[Batch]:
    return list(Batch(engine=self.engine, setting=self.setting, ...))


for batch in self.get_barches(): 
     batch_results = await Batch.perform()
     parsed_result = Parse(batch_result)

ManiMozaffar avatar Aug 19 '23 13:08 ManiMozaffar