awesome-crawler
awesome-crawler copied to clipboard
BruceDone
A collection of awesome web crawler,spider in different languages
I might be biased about this library, but I think it's worth being on the list! 😅
A program to download photo and video from [any site](https://github.com/AAndyProgram/SCrawler#supported-sites) (e.g. Reddit, Twitter, Instagram). https://github.com/AAndyProgram/SCrawler
Link to markdown: https://gist.github.com/gustavorps/98a9b03143db24c5acedab343b7e7a55 # Preview 
Here is my crawler made in Rust. Supports seed database, adds external links to database, simple parsing and adding to elastic search. https://github.com/tclaus/web_crawler
Add Internet Archive [Brozzler](https://github.com/internetarchive/brozzler) distributed crawler in Python
Is there a crawler with an web GUI or rather a frontend such as "Octoparse" for Windows oder "Web Alert" for Android?
Added [MicroFrontier](https://github.com/adileo/MicroFrontier) to the list.
