scrapy-mongodb icon indicating copy to clipboard operation
scrapy-mongodb copied to clipboard

Published 20 hours ago • verified publisher icon sebdah

Is the pipeline blocking?

Open jxltom opened this issue 8 years ago • 0 comments

It looks like the scrapy-mongodb pipeline is blocking. So the crawl speed is limited to the mongodb data write speed.

Is it better to change it to non-blocking pipeline such as using txmongo?

jxltom avatar Sep 24 '17 04:09 jxltom