scrapyrt icon indicating copy to clipboard operation
scrapyrt copied to clipboard

Requesting max_items in scrapyrt

Open xaander1 opened this issue 5 years ago • 2 comments
trafficstars

max_items like the max_requests argument would be really helpful.

xaander1 avatar May 05 '20 19:05 xaander1

To do this look up how max_requests are done now: https://github.com/scrapinghub/scrapyrt/blob/master/scrapyrt/core.py#L99 and add it in a similar way, need to count items collected somewhere here: https://github.com/scrapinghub/scrapyrt/blob/master/scrapyrt/core.py#L207 and close spider if it exceeds limits.

pawelmhm avatar Jan 29 '21 12:01 pawelmhm

assigned to @serhii73

pawelmhm avatar May 31 '21 14:05 pawelmhm