crawling-framework topic

List of libraries, tools and APIs for web scraping and data processing.

lorien

🌌 High productivity semi-automatic crawler generator 🛠️🧰

rollrat

Web Scan Lazy Tools - Python Package

Symbo1

Async Python 3.6+ web scraping micro-framework based on asyncio

howie6879

Crawlzone is a fast asynchronous internet crawling framework for PHP.

crawlzone

Vietnamese text data crawler scripts for various sites (including Youtube, Facebook, 4rum, news, ...)

behitek

Serritor is an open source web crawler framework built upon Selenium and written in Java. It can be used to crawl dynamic web pages that require JavaScript to render data.

peterbencze

🔍 A powerful web-crawling framework, based on aiohttp.

wind2sing

Easily crawl news portals or blog sites using Storm Crawler.

tokenmill

A Python framework to build polite, but tenacious crawlers / scrapers with a MariaDB backend

RuedigerVoigt