googlebot topic
List
googlebot repositories
ipranges
419
Stars
76
Forks
Watchers
🔨 List all IP ranges from: Google (Cloud & GoogleBot), Bing (Bingbot), Amazon (AWS), Microsoft, Oracle (Cloud), GitHub, Facebook (Meta), OpenAI (GPTBot) and other with daily updates.
robots.txt
83
Stars
37
Forks
Watchers
Simple robots.txt template. Keep unwanted robots out (disallow). White lists (allow) legitimate user-agents. Useful for all websites.
yzhanIP
58
Stars
3
Forks
Watchers
Crawl, match, parse IP or IP range, check if IP or range is in another range. Support IPv4, IPv6, IP Interval, Wildcard and CIDR. Check if IP is Cloudflare node IP, Google bot IP. 爬取，正则匹配，解析...
legitbot
19
Stars
9
Forks
Watchers
🤔 Is this Web request from a real search engine🕷 or from an impersonating agent 🕵️♀️?