web-crawling topic

List web-crawling repositories

daenerys

63
Stars
30
Forks
Watchers

Scraping and Web Crawling Framework For Zhihu Live

scrapyd-cluster-on-heroku

122
Stars
94
Forks
Watchers

Set up free and scalable Scrapyd cluster for distributed web-crawling with just a few clicks. DEMO :point_right:

crawlee

14.3k
Stars
597
Forks
99
Watchers

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and o...

InfinityCrawler

239
Stars
35
Forks
Watchers

A simple but powerful web crawler library for .NET

Bet-on-Sibyl

243
Stars
93
Forks
Watchers

Machine Learning Model for Sport Predictions (Football, Basketball, Baseball, Hockey, Soccer & Tennis)

krawler

130
Stars
16
Forks
Watchers

A web crawling framework written in Kotlin

scrapy-training

170
Stars
46
Forks
Watchers

Scrapy Training companion code

amazon-scraper

302
Stars
155
Forks
Watchers

A simple web scraper to extract Product Data and Pricing from Amazon

ayakashi

198
Stars
8
Forks
Watchers

:zap: Ayakashi.io - The next generation web scraping framework

bancocentralbrasil

120
Stars
34
Forks
Watchers

💵 💰 :brazil: Informações sobre taxas oficiais diárias de Inflação, Selic, Poupança, Dólar, Dólar PTAX, Euro e Euro PTAX pelo site do Banco Central do Brasil