web-robots topic

List web-robots repositories

Stars

Forks

Watchers

Simple robots.txt template. Keep unwanted robots out (disallow). White lists (allow) legitimate user-agents. Useful for all websites.