Number1Tao
Number1Tao
检查合并索引文件，是否内存溢出？处理速度太慢，10万条新闻的索引合并需要16小时。。。。
``` def parse(self,response): if self.web_id>self.crawl_number: if self.has_terminated==False: self.write_block_data.close() self.write_block_crawlwd_weburl.close() del self.write_block_crawlwd_weburl,self.write_block_data self.has_terminated==True ``` 2016-05-22 01:26:58 [scrapy] ERROR: Spider error processing (referer: http://news.163.com/) Traceback (most recent call last): File "c:\python27\lib\site-packages\scrapy\utils\defer.py", line...