distributed-crawler topic
List
distributed-crawler repositories
supercrawler
371
Stars
62
Forks
Watchers
A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.
weibospider
4.8k
Stars
1.2k
Forks
Watchers
:zap: A distributed crawler for weibo, building with celery and requests.
Distributed-Multi-User-Scrapy-System-with-a-Web-UI
110
Stars
45
Forks
Watchers
Django based application that allows creating, deploying and running Scrapy spiders in a distributed manner
DCVS
23
Stars
5
Forks
Watchers
🎓 JD Distributed Crawler and Visualization System. 京东商品评论分布式爬虫
Ugly-Distributed-Crawler
46
Stars
14
Forks
Watchers
基于Redis实现的简单到爆的分布式爬虫