commoncrawl topic

Inspired by google c4, here is a series of colossal clean data cleaning scripts focused on CommonCrawl data processing. Including Chinese data processing and cleaning methods in MassiveText.

shjwudp

Extract web archive data using Wayback Machine and Common Crawl

karust

来自[码云](https://gitee.com/generals-space/site-mirror-go) 通用爬虫, 仿站工具, 整站下载

generals-space

Word analysis, by domain, on the Common Crawl data set for the purpose of finding industry trends

CI-Research

A News Article Collection Library

networkdynamics

A very simple news crawler with a funny name

flairNLP

Python tools to retrieve text from CommonCrawl WARC files based on cdx index.

lxucs