EasySpider
EasySpider copied to clipboard
希望程序可以在发起http请求前判断url是否正常
错误log打印如下,看上去是在下载url为空的资源,但是我并没有头绪到底哪里有问题。
希望程序可以在发起http请求的时候可以提前判断请求的地址是否正常,如果不正常能跳过这个资源并打印相关信息,这样至少可以避免崩溃。
-------Retrying-------
Exception in thread Thread-4:
Traceback (most recent call last):
File "easyspider_executestage.py", line 477, in loopExecute
File "easyspider_executestage.py", line 385, in executeNode
File "easyspider_executestage.py", line 522, in loopExecute
File "easyspider_executestage.py", line 371, in executeNode
File "easyspider_executestage.py", line 969, in getData
File "easyspider_executestage.py", line 840, in get_content
File "easyspider_executestage.py", line 48, in download_image
File "requests\api.py", line 73, in get
File "requests\api.py", line 59, in request
File "requests\sessions.py", line 573, in request
File "requests\sessions.py", line 484, in prepare_request
File "requests\models.py", line 368, in prepare
File "requests\models.py", line 439, in prepare_url
requests.exceptions.MissingSchema: Invalid URL '': No scheme supplied. Perhaps you meant http://?
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "threading.py", line 1038, in _bootstrap_inner
File "easyspider_executestage.py", line 167, in run
File "easyspider_executestage.py", line 362, in executeNode
File "easyspider_executestage.py", line 385, in executeNode
File "easyspider_executestage.py", line 499, in loopExecute
File "easyspider_executestage.py", line 385, in executeNode
File "easyspider_executestage.py", line 522, in loopExecute
File "easyspider_executestage.py", line 371, in executeNode
File "easyspider_executestage.py", line 969, in getData
File "easyspider_executestage.py", line 840, in get_content
File "easyspider_executestage.py", line 48, in download_image
File "requests\api.py", line 73, in get
File "requests\api.py", line 59, in request
File "requests\sessions.py", line 573, in request
File "requests\sessions.py", line 484, in prepare_request
File "requests\models.py", line 368, in prepare
File "requests\models.py", line 439, in prepare_url
requests.exceptions.MissingSchema: Invalid URL '': No scheme supplied. Perhaps you meant http://?
Cannot automatically check new version, please use the following command to check whether a new version avaliable and upgrade by pip:
[1;32mpip index versions commandline_config
pip install commandline --upgrade[0m
Thread with task id: 31 is closed
你的任务流程是什么,我不懂你想做什么,可以把你的任务文件发给我看看:[email protected]
Bug已修复,下个版本可用。