python-D

Results 2 issues of python-D

2019-04-03 12:25:25 [scrapy.extensions.logstats] INFO: Crawled 18852 pages (at 0 pages/min), scraped 1762 items (at 0 items/min)

2 comment

爬虫运行过程中会出现2019-04-03 12:25:25 [scrapy.extensions.logstats] INFO: Crawled 18852 pages (at 0 pages/min), scraped 1762 items (at 0 items/min)这种状况，大概过了一到两分钟才重新拉取到redis的队列里面的url

bug
question
improvement

@app.before_first_request

1 comment

这个方法会导致一段时间以后Chrome.py进程自动挂掉