crawling topic
crawler
🕷️ An easy-to-use spider written in Golang. (previous named GOPA.)
double-agent
A test suite of common scraper detection techniques. See how detectable your scraper stack is.
proxifier
A fast, modern and intelligent proxy rotator perfect for crawling and scraping public data.
Harvester
Web crawling and document processing through a usable interface.
pomp
Screen scraping and web crawling framework
talospider
talospider - A simple,lightweight scraping micro-framework
wget-lua
Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.
robots.txt
Simple robots.txt template. Keep unwanted robots out (disallow). White lists (allow) legitimate user-agents. Useful for all websites.
telegram-crawler
🕷 Automatically detect changes made to the official Telegram sites, clients and servers.