网络爬虫

[sdcuike]

java

https://github.com/jhy/jsoup/

https://github.com/internetarchive/heritrix3 https://github.com/code4craft/webmagic

https://github.com/brianway/webporter

http://stormcrawler.net/ https://github.com/DigitalPebble/storm-crawler http://nutch.apache.org/ https://github.com/yahoo/anthelion

他人整理

https://github.com/BruceDone/awesome-crawler

Name Language Platform Heritrix Java Linux Nutch Java Cross-platform Scrapy Python Cross-platform DataparkSearch C++ Cross-platform GNU Wget C Linux GRUB C#, C, Python, Perl Cross-platform ht://Dig C++ Unix HTTrack C/C++ Cross-platform ICDL Crawler C++ Cross-platform mnoGoSearch C Windows Norconex HTTP Collector Java Cross-platform Open Source Server C/C++, Java PHP Cross-platform PHP-Crawler PHP Cross-platform YaCy Java Cross-platform WebSPHINX Java Cross-platform WebLech Java Cross-platform Arale Java Cross-platform JSpider Java Cross-platform HyperSpider Java Cross-platform Arachnid Java Cross-platform Spindle Java Cross-platform Spider Java Cross-platform LARM Java Cross-platform Metis Java Cross-platform SimpleSpider Java Cross-platform Grunk Java Cross-platform CAPEK Java Cross-platform Aperture Java Cross-platform Smart and Simple Web Crawler Java Cross-platform Web Harvest Java Cross-platform Aspseek C++ Linux Bixo Java Cross-platform crawler4j Java Cross-platform Ebot Erland Linux Hounder Java Cross-platform Hyper Estraier C/C++ Cross-platform OpenWebSpider C#, PHP Cross-platform Pavuk C Lunix Sphider PHP Cross-platform Xapian C++ Cross-platform Arachnode.net C# Windows Crawwwler C++ Java Distributed Web Crawler C, Java, Python Cross-platform iCrawler Java Cross-platform pycreep Java Cross-platform Opese C++ Linux Andjing Java Ccrawler C# Windows WebEater Java Cross-platform JoBo Java Cross-platfor