A Web crawler, sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web and that is typically operated by search engines for the purpose of Web indexing (web spidering).
Python script to check if there is any differences in responses of an application when the request comes from a search engine's crawler.
Incredibly fast crawler designed for OSINT.
A Swift Web Crawler 🕷
Scrape data from Arena of Valor's official website.
Scrapes all photos and videos in a web page / Instagram / Twitter / Tumblr / Reddit / pixiv / TikTok
Fast, highly configurable, cloud native dark web crawler.
Kotlin library, Validator box that can inspect any type of form, provides multiple validation functions with an inclusion of clearing views
Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more