robots-exclusion-standard topic

NodeJS robots.txt parser with support for wildcard (*) matching.

samclarke

Simple robots.txt template. Keep unwanted robots out (disallow). White lists (allow) legitimate user-agents. Useful for all websites.

jonasjacek

Go robots.txt parser

samclarke