robotstxt topic

A native Rust port of Google's robots.txt parser and matcher C++ library.

Folyd

Be nice on the web

dmi3kno

robots.txt file parsing and checking for R

ropensci

Generator robots.txt for node js

itgalaxy

Makes it easy to add robots.txt, sitemap and web app manifest during build to your Astro app.

alextim

A webpack plugin to generate a robots.txt file

itgalaxy

Go robots.txt parser

samclarke