scrapy-splash icon indicating copy to clipboard operation
scrapy-splash copied to clipboard

Misleading error message with SPLASH_URL missing scheme

Open redapple opened this issue 7 years ago • 0 comments

Originally appeared on StackOverflow.

When SPLASH_URL setting is missing the http:// scheme, scrapy's error is not very helpful

SPLASH_URL = 'localhost:8050'
(...)
2017-06-01 14:44:35 [scrapy.core.scraper] ERROR: Error downloading <GET http://example.com/ via http://example.com/>
Traceback (most recent call last):
  File "/home/paul/.virtualenvs/scrapy-splash/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1301, in _inlineCallbacks
    result = g.send(result)
  File "/home/paul/.virtualenvs/scrapy-splash/local/lib/python2.7/site-packages/scrapy/core/downloader/middleware.py", line 37, in process_request
    response = yield method(request=request, spider=spider)
  File "/home/paul/.virtualenvs/scrapy-splash/local/lib/python2.7/site-packages/scrapy_splash/middleware.py", line 358, in process_request
    priority=request.priority + self.rescheduling_priority_adjust
  File "/home/paul/.virtualenvs/scrapy-splash/local/lib/python2.7/site-packages/scrapy/http/request/__init__.py", line 94, in replace
    return cls(*args, **kwargs)
  File "/home/paul/.virtualenvs/scrapy-splash/local/lib/python2.7/site-packages/scrapy_splash/request.py", line 76, in __init__
    **kwargs)
  File "/home/paul/.virtualenvs/scrapy-splash/local/lib/python2.7/site-packages/scrapy/http/request/__init__.py", line 25, in __init__
    self._set_url(url)
  File "/home/paul/.virtualenvs/scrapy-splash/local/lib/python2.7/site-packages/scrapy/http/request/__init__.py", line 58, in _set_url
    raise ValueError('Missing scheme in request url: %s' % self._url)
ValueError: Missing scheme in request url: render.html

Maybe the plugin can detect that the splash URL is wrong and fail with a more actionable message.

redapple avatar Jun 01 '17 12:06 redapple