milkeasd

Is there a way to stop spider check duplicate with redis ?

7

My spider was extremely slow when run with scrapy-redis. Because there is a big delay between slave and master. I want to reduce the commuication to just only getting the...

My scrapyd output files sudden change to /root

1

I run my spider in a scrapyd linux server. As usual, I use the following feeds config. ``` 'FEEDS': { 'items.csv': { 'format': 'csv', 'encoding': 'utf8', } ``` But, it...