speed

The URI you submitted has disallowed characters.

3

http://www.lankadeepa.lk/latest_news/ශ්‍රී-ලන්කන්-ගුවන්-සේවය-ප්‍රතිසංවිධානයට-ජනපතිගෙන්-කමිටුවක්/1-543320 An Error Was Encountered The URI you submitted has disallowed characters.

max_allowed_packet

1

2017-06-28 09:48:59,005--ERROR --Save data failure, url:https://www.pornhub.com/view_video.php?viewkey=ph56c925e408be4, log:javax.jdo.JDODataStoreException: Insert of object "com.soso.crawler.model.Crawler_CrawlDataTable@58a70d1" using statement "INSERT INTO `CRAWLER_CRAWLDATATABLE` (`FETCHTIMES`,`SITEID`,`DEPLOYLOG`,`STATUS`,`MD5DIGEST`,`CRAWLURL`,`CRAWLURLID`,`ORDERINDEX`,`CHECKLOG`,`CREATEDATE`,`DATA`,`MD5DIGESTHASH`) VALUES (?,?,?,?,?,?,?,?,?,?,?,?)" failed : Packet for query is too large (1764526 > 1048576)....

增加数据采集日报发送功能

2

可以推送到指定邮箱

更多的选项配置： 正则过滤 默认值 必须（不能为空） ......

目前采集节点需要在WEB端手动添加，可以实现采集节点自动上线下线功能。 这样通过Docker部署爬虫集群时会非常方便管理

采集规则更新后增加发布动作，默认发布按钮是灰色，只有在更新才可操作 采集规则更新后不实时生效，需要发布才生效