web-crawler topic

Explore a website recursively and download all the wanted documents (PDF, ODT…)

Siltaar

Interactive CLI Web Crawler

saeeddhqan

A utility for crawling websites and building frequency lists of words

calebwin

基于C#.NET+PhantomJS+Sellenium的高级网络爬虫程序。可执行Javascript代码、触发各类事件、操纵页面Dom结构。

microfisher

A collection of awesome web scaper, crawler.

duyet

Dyer is designed for reliable, flexible and fast web crawling, providing some high-level, comprehensive features without compromising speed.

hominee

Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

sjdirect

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and o...

apify

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

crawlab-team

新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

ssssssss-team