docsearch-scraper icon indicating copy to clipboard operation
docsearch-scraper copied to clipboard

Published 20 hours ago • verified publisher icon algolia

concurrency settings

Open davidejones opened this issue 4 years ago • 1 comments
trafficstars

As you might imagine while our content has grown so too has the time to index. Is there anyway the concurrency settings of scrapy could be exposed? or some feature to run sections of an index in parallel be added to a roadmap?

e.g settings like this

REACTOR_THREADPOOL_MAXSIZE = 128
CONCURRENT_REQUESTS = 256
CONCURRENT_REQUESTS_PER_DOMAIN = 256
CONCURRENT_REQUESTS_PER_IP = 256

davidejones avatar Jul 09 '21 14:07 davidejones

Yes, please. Is there any concurrency currently? If not, there really should be...

williamblevins avatar Oct 18 '21 15:10 williamblevins