spiderman

spiderman copied to clipboard

爬虫运行一段时间后报错如下，然后就中断无法运行了 Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/scrapy/core/downloader/middleware.py", line 44, in process_request return (yield download_func(request=request, spider=spider)) twisted.web._newclient.ResponseNeverReceived: [<twisted.python.failure.Failure twisted.internet.error.ConnectionDone: Connection was closed cleanly.>]

看起来是网络问题或者目标网站的问题