scrapyrt
scrapyrt copied to clipboard
Requesting max_items in scrapyrt
trafficstars
max_items like the max_requests argument would be really helpful.
To do this look up how max_requests are done now: https://github.com/scrapinghub/scrapyrt/blob/master/scrapyrt/core.py#L99 and add it in a similar way, need to count items collected somewhere here: https://github.com/scrapinghub/scrapyrt/blob/master/scrapyrt/core.py#L207 and close spider if it exceeds limits.
assigned to @serhii73