crawlers topic

Python script to check if there is any differences in responses of an application when the request comes from a search engine's crawler.

p0dalirius

基于python的网页自动化工具。既能控制浏览器，也能收发数据包。可兼顾浏览器自动化的便利性和requests的高效率。功能强大，内置无数人性化设计和便捷功能。语法简洁而优雅，代码量少。

g1879

Powerful dork searcher and vulnerability scanner for windows platform

ph09nix

An R web crawler and scraper

salimk

🤖/👨‍🦰 Detect bots/crawlers/spiders using the user agent string

omrilotan

hproxy - Asynchronous IP proxy pool, aims to make getting proxy as convenient as possible.(异步爬虫代理池)

howie6879

Proxy List Scrapper

narkhedesam

Vietnamese text data crawler scripts for various sites (including Youtube, Facebook, 4rum, news, ...)

behitek

Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.

ArchiveTeam

Simple robots.txt template. Keep unwanted robots out (disallow). White lists (allow) legitimate user-agents. Useful for all websites.

jonasjacek