Crawler topic

A Web crawler, sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web and that is typically operated by search engines for the purpose of Web indexing (web spidering).

List Crawler repositories

crawlersuseragents

19
Stars
3
Forks
Watchers

Python script to check if there is any differences in responses of an application when the request comes from a search engine's crawler.

Photon

10.2k
Stars
1.5k
Forks
Watchers

Incredibly fast crawler designed for OSINT.

Selenops

252
Stars
18
Forks
Watchers

A Swift Web Crawler 🕷

arena-of-valor-data-scraper

9
Stars
2
Forks
Watchers

Scrape data from Arena of Valor's official website.

media-scraper

348
Stars
49
Forks
Watchers

Scrapes all photos and videos in a web page / Instagram / Twitter / Tumblr / Reddit / pixiv / TikTok

bathyscaphe

89
Stars
23
Forks
Watchers

Fast, highly configurable, cloud native dark web crawler.

crawler-jsoup-maven

46
Stars
24
Forks
Watchers

This is a crawler(reptile)

AndroidValidatorCrawler

23
Stars
4
Forks
Watchers

Kotlin library, Validator box that can inspect any type of form, provides multiple validation functions with an inclusion of clearing views

cariddi

1.2k
Stars
141
Forks
Watchers

Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more

slrp

136
Stars
21
Forks
Watchers

rotating open proxy multiplexer