Crawler topic
A Web crawler, sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web and that is typically operated by search engines for the purpose of Web indexing (web spidering).
crawlersuseragents
Python script to check if there is any differences in responses of an application when the request comes from a search engine's crawler.
Photon
Incredibly fast crawler designed for OSINT.
Selenops
A Swift Web Crawler 🕷
arena-of-valor-data-scraper
Scrape data from Arena of Valor's official website.
media-scraper
Scrapes all photos and videos in a web page / Instagram / Twitter / Tumblr / Reddit / pixiv / TikTok
bathyscaphe
Fast, highly configurable, cloud native dark web crawler.
AndroidValidatorCrawler
Kotlin library, Validator box that can inspect any type of form, provides multiple validation functions with an inclusion of clearing views
cariddi
Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more