warc topic

A small tool which uses the CommonCrawl URL Index to download documents with certain file types or mime-types. This is used for mass-testing of frameworks like Apache POI and Apache Tika

centic9

🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...

ArchiveBox

Serverless replay of web archives directly in the browser

webrecorder

News crawling with StormCrawler - stores content as WARC

commoncrawl

Bitextor generates translation memories from multilingual websites

bitextor

WarcDB: Web crawl data as SQLite databases.

Florents-Tselai

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

ArchiveTeam

Collect and revisit web pages.

Rhizome-Conifer

:whale2: Web Archiving Integration Layer: One-Click User Instigated Preservation

machawk1

InterPlanetary Wayback: A distributed and persistent archive replay system using IPFS

oduwsdl