sergeospb

failed to crawl

1

python crawler.py --mirror --url "http://google.com/" 2013-03-23 09:26:46+0400 [scrapy] INFO: Scrapy 0.17.0 started (bot: crawltest) 2013-03-23 09:26:46+0400 [scrapy] DEBUG: Overridden settings: {'NEWSPIDER_MODULE': 'crawltest.spiders', 'SPIDER_MODULES': ['crawltest.spiders'], 'DOWNLOADER_HTTPCLIENTFACTORY': 'warcclientfactory.WarcHTTPClientFactory', 'BOT_NAME': 'crawltest'} 2013-03-23 09:26:46+0400...