crawlers topic

Norconex Crawlers (or spiders) are flexible web and filesystem crawlers for collecting, parsing, and manipulating data from the web or filesystem to various data repositories such as search engines.

Norconex

Open source SEO auditing tool.

StJudeWasHere

Serritor is an open source web crawler framework built upon Selenium and written in Java. It can be used to crawl dynamic web pages that require JavaScript to render data.

peterbencze

User agent database in JSON format of bots, crawlers, certain malware, automated software, scripts and uncommon ones.

herrbischoff

Licitações de Feira de Santana de fácil acesso aos cidadãos 🏦

anapaulagomes

Block crawlers and high traffic users on your site by IP using Redis

Potelo

Sneakpeek is a framework that helps to quickly and conviniently develop scrapers. It’s the best choice for scrapers that have some specific complex scraping logic that needs to be run on a constant ba...

flulemon

Tiny script to crawl information of a specific application in the Google play/store base on PHP.

BaseMax

An open source web crawling platform

zcrawl

A list of AI agents and robots to block.

ai-robots-txt