scrapy-pyppeteer icon indicating copy to clipboard operation
scrapy-pyppeteer copied to clipboard

Published 20 hours ago • verified publisher icon elacuesta

How to control concurrency?

Open luzihang123 opened this issue 3 years ago • 1 comments

demand:10W web pages rendered into Base64 upload OSS one day

luzihang123 avatar Jul 10 '20 00:07 luzihang123

First of all, thanks for taking an interest in this project.

Concurrency should be handled by the Scrapy Scheduler (which acts before the handler provided by this package), taking into account settings like CONCURRENT_REQUESTS, CONCURRENT_REQUESTS_PER_DOMAIN and CONCURRENT_REQUESTS_PER_IP. Could you elaborate a bit more on you use case, maybe with some minimal code to help me understand the scenario? Thanks again.

elacuesta avatar Jul 10 '20 14:07 elacuesta