scrapy-splash
scrapy-splash copied to clipboard
Misleading error message with SPLASH_URL missing scheme
Originally appeared on StackOverflow.
When SPLASH_URL
setting is missing the http://
scheme, scrapy's error is not very helpful
SPLASH_URL = 'localhost:8050'
(...)
2017-06-01 14:44:35 [scrapy.core.scraper] ERROR: Error downloading <GET http://example.com/ via http://example.com/>
Traceback (most recent call last):
File "/home/paul/.virtualenvs/scrapy-splash/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1301, in _inlineCallbacks
result = g.send(result)
File "/home/paul/.virtualenvs/scrapy-splash/local/lib/python2.7/site-packages/scrapy/core/downloader/middleware.py", line 37, in process_request
response = yield method(request=request, spider=spider)
File "/home/paul/.virtualenvs/scrapy-splash/local/lib/python2.7/site-packages/scrapy_splash/middleware.py", line 358, in process_request
priority=request.priority + self.rescheduling_priority_adjust
File "/home/paul/.virtualenvs/scrapy-splash/local/lib/python2.7/site-packages/scrapy/http/request/__init__.py", line 94, in replace
return cls(*args, **kwargs)
File "/home/paul/.virtualenvs/scrapy-splash/local/lib/python2.7/site-packages/scrapy_splash/request.py", line 76, in __init__
**kwargs)
File "/home/paul/.virtualenvs/scrapy-splash/local/lib/python2.7/site-packages/scrapy/http/request/__init__.py", line 25, in __init__
self._set_url(url)
File "/home/paul/.virtualenvs/scrapy-splash/local/lib/python2.7/site-packages/scrapy/http/request/__init__.py", line 58, in _set_url
raise ValueError('Missing scheme in request url: %s' % self._url)
ValueError: Missing scheme in request url: render.html
Maybe the plugin can detect that the splash URL is wrong and fail with a more actionable message.