crawl topic
List
crawl repositories
grab-site
1.3k
Stars
125
Forks
Watchers
The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns
laravel-site-search
270
Stars
20
Forks
Watchers
Create a full-text search index by crawling your site
Pinkerton
280
Stars
46
Forks
Watchers
🕵️ Pinkerton is an JavaScript file crawler and secret finder tool developed in Python
dl_coursera
140
Stars
30
Forks
Watchers
A simple, fast, and reliable Coursera crawling & downloading tool
Scrapy_IPProxyPool
101
Stars
41
Forks
Watchers
免费 IP 代理池。Scrapy 爬虫框架插件
WebSecurityArticles
78
Stars
19
Forks
Watchers
爬取及整理Freebuf\安全客\先知\知道创宇等站点的”web安全“类优质文章
wget-lua
83
Stars
14
Forks
Watchers
Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.