robots-txt topic
gatsby-plugin-robots-txt
Gatsby plugin that automatically creates robots.txt for your site
advertools
advertools - online marketing productivity and analysis tools
robotstxt
The robots.txt exclusion protocol implementation for Go language
crawler-commons
A set of reusable Java components that implement functionality common to any web crawler
InfinityCrawler
A simple but powerful web crawler library for .NET
ultimate-sitemap-parser
Ultimate Website Sitemap Parser
fetchbot
A simple and flexible web crawler that follows the robots.txt policies and crawl delays.
gocrawl
Polite, slim and concurrent web crawler.
robots
NuxtJS module for robots.txt
robots-txt
Determine if a page may be crawled from robots.txt, robots meta tags and robot headers