web-scraping topic

List web-scraping repositories

Humanoid

202
Stars
27
Forks
Watchers

Node.js package to bypass CloudFlare's anti-bot JavaScript challenges

halfstaff

22
Stars
0
Forks
Watchers

:us: Is the US flag at half-staff?

DAT8

1.6k
Stars
1.1k
Forks
Watchers

General Assembly's 2015 Data Science course in Washington, DC

fb_friend_list_scraper

229
Stars
22
Forks
Watchers

OSINT tool to scrape names and usernames from large friend lists on Facebook, without being rate limited.

arachnid

77
Stars
12
Forks
Watchers

Powerful web scraping framework for Crystal

snoop

2.7k
Stars
325
Forks
Watchers

Snoop — инструмент разведки на основе открытых данных (OSINT world)

scrapy-wayback-machine

107
Stars
28
Forks
Watchers

A Scrapy middleware for scraping time series data from Archive.org's Wayback Machine.

htmldate

113
Stars
27
Forks
Watchers

Fast and robust date extraction from web pages, with Python or on the command-line

trafilatura

3.0k
Stars
228
Forks
Watchers

Python & command-line tool to gather text on the Web: web crawling/scraping, extraction of text, metadata, comments

dude

412
Stars
20
Forks
Watchers

dude uncomplicated data extraction: A simple framework for writing web scrapers using Python decorators