scrapy-selenium icon indicating copy to clipboard operation
scrapy-selenium copied to clipboard

Published 20 hours ago • verified publisher icon clemfromspace

How to do concurrent scraping?

Open vbuterin2 opened this issue 3 years ago • 2 comments

I currently use single thread scraper to crawl google.com. But I have tons of search terms. scrapy-selenium only open 1 browser, so I could only search one term at one time. Should I use remote browser or other solutions?

vbuterin2 avatar Jun 01 '21 18:06 vbuterin2

@vbuterin2 I think your question is a duplicate of https://github.com/clemfromspace/scrapy-selenium/issues/76

Flushot avatar Jul 09 '21 04:07 Flushot

You could run two scrapy crawl spider_name processes and set CONCURRENT_REQUESTS = 1.

cs246810 avatar Jul 14 '21 13:07 cs246810