robots-parser topic

NodeJS robots.txt parser with support for wildcard (*) matching.

samclarke

An extensible robots.txt parser and client library, with full support for every directive and specification.

VIPnytt

robots.txt parser for Rust.

messense

A pure-Python robots.txt parser with support for modern conventions.

scrapy

Go robots.txt parser

samclarke

:robot: robots.txt as a service. Crawls robots.txt files, downloads and parses them to check rules through an API

fooock