books_crawler
books_crawler copied to clipboard
No module name scrapy_magicfields
While running the script as per instructions available on this blog post INCREMENTAL CRAWLS WITH SCRAPY AND DELTAFETCH, I got this error No module name scrapy_magicfields.
Full output of error:
Unhandled error in Deferred:
2017-08-13 22:02:52 [twisted] CRITICAL: Unhandled error in Deferred:
2017-08-13 22:02:52 [twisted] CRITICAL:
Traceback (most recent call last):
File "/home/kapil/Projects/scrape_techchrunch/venv/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1386, in _inlineCallbacks
result = g.send(result)
File "/home/kapil/Projects/scrape_techchrunch/venv/local/lib/python2.7/site-packages/scrapy/crawler.py", line 95, in crawl
six.reraise(*exc_info)
File "/home/kapil/Projects/scrape_techchrunch/venv/local/lib/python2.7/site-packages/scrapy/crawler.py", line 77, in crawl
self.engine = self._create_engine()
File "/home/kapil/Projects/scrape_techchrunch/venv/local/lib/python2.7/site-packages/scrapy/crawler.py", line 102, in _create_engine
return ExecutionEngine(self, lambda _: self.stop())
File "/home/kapil/Projects/scrape_techchrunch/venv/local/lib/python2.7/site-packages/scrapy/core/engine.py", line 70, in __init__
self.scraper = Scraper(crawler)
File "/home/kapil/Projects/scrape_techchrunch/venv/local/lib/python2.7/site-packages/scrapy/core/scraper.py", line 69, in __init__
self.spidermw = SpiderMiddlewareManager.from_crawler(crawler)
File "/home/kapil/Projects/scrape_techchrunch/venv/local/lib/python2.7/site-packages/scrapy/middleware.py", line 58, in from_crawler
return cls.from_settings(crawler.settings, crawler)
File "/home/kapil/Projects/scrape_techchrunch/venv/local/lib/python2.7/site-packages/scrapy/middleware.py", line 34, in from_settings
mwcls = load_object(clspath)
File "/home/kapil/Projects/scrape_techchrunch/venv/local/lib/python2.7/site-packages/scrapy/utils/misc.py", line 44, in load_object
mod = import_module(module)
File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
ImportError: No module named scrapy_magicfields
But after installing scrapy_magicfields
with pip
, it solved the issue. But there was no mention about need to install this module as well.