ai-scraping topic

The fastest web crawler written in Rust. Maintained by @a11ywatch.

spider-rs

Python scraper based on AI

ScrapeGraphAI

🔥 The Web Data API for AI - Turn entire websites into LLM-ready markdown or structured data

firecrawl

🔥 This repository contains complete application examples, including websites and other projects, developed using Firecrawl.

firecrawl

A Powerful web scraper powered by LLM | OpenAI, Gemini & Ollama

itsOwen

Crawl a website starting from a URL, find relevant pages, and extract data – all guided by your natural language prompt.

oxylabs

🕷️ An undetectable, powerful, flexible, high-performance Python library to make Web Scraping Easy and Effortless as it should be!

D4Vinci

AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.

any4ai

Structured data gathering from any website using AI-powered scraper, crawler, and browser automation. Scraping and crawling with natural language prompts. Equip your LLM agents with fresh data. AI Stu...

oxylabs

Lightweight library for scraping web-sites with LLMs

raznem