web-crawling topic

Simple robots.txt template. Keep unwanted robots out (disallow). White lists (allow) legitimate user-agents. Useful for all websites.

jonasjacek

Parser and database to index the terpene profile of different strains of Cannabis from online databases

MaxValue

Library for Rapid (Web) Crawler and Scraper Development

crwlrsoft

Command Line Tool to download torrents

alyakhtar

Web Scraping Craigslist's Engineering Jobs in NY with Scrapy

GoTrained

Web Scraping Framework

lorien

Amazon products scraper with using of rotating proxies and headless Chrome from ScrapingAnt

ScrapingAnt

Compares price of the product entered by the user from e-commerce sites Amazon and Flipkart :moneybag: :bar_chart:

sushantPatrikar

CrawlerX - Develop Extensible, Distributed, Scalable Crawler System which is a web platform that can be used to crawl URLs in different kind of protocols in a distributed way.

leopardslab

JAW: A Graph-based Security Analysis Framework for Client-side JavaScript

SoheilKhodayari