devxiaosong comments

Repositories
Issues
Comments

Results 4 comments of


                                            devxiaosong

Hard to visit the Timer Tasks page when there are lots of history

👌 手工删除日志数据。还有一部分数据，抓取的数据统计图（progress visualization）的数据是保存在哪里的？没找到。

Hard to visit the Timer Tasks page when there are lots of history

BACKUP_STATS_JSON_FILE to False ，设置后是完全不保留统计数据，还是有个定时清理机制？

scrapydweb down due to use out of memory

Of course, logparser is running on each side. If it is not running on scrapydweb server, what does happen? The server fetchs log from scrapy node server?

How to deploy distributed spiders based on scrapy-redis

对于分布式爬虫这块，我的理解，现在的方式还是需要redis客户端发送启动命令。能在scrapydweb里直接实现，那就最好了。