robots-txt topic

NodeJS robots.txt parser with support for wildcard (*) matching.

samclarke

Simple robots.txt template. Keep unwanted robots out (disallow). White lists (allow) legitimate user-agents. Useful for all websites.

jonasjacek

Open-Source Python Based SEO Web Crawler

beb7

Php class for robots.txt parse

t1gor

Parser for robots.txt for node.js

ekalinin

Generator robots.txt for node js

itgalaxy

grobotstxt is a native Go port of Google's robots.txt parser and matcher library.

jimsmart

.Net Core Plugin Manager, extend web applications using plugin technology enabling true SOLID and DRY principles when developing applications

k3ldar

Makes it easy to add robots.txt, sitemap and web app manifest during build to your Astro app.

alextim

An extensible robots.txt parser and client library, with full support for every directive and specification.

VIPnytt