robots-txt topic

Dark Web Informationgathering Footprinting Scanner and Recon Tool Release. Dark Web is an Information Gathering Tool I made in python 3. To run Dark Web, it only needs a domain or ip. Dark Web can wo...

akashblackhat

A python script to check if URLs are allowed or disallowed by a robots.txt file.

p0dalirius

This package helps you to add meta-tags, sitemap.xml and robots.txt into your project easily.

EngincanV

- THIS IS AN OLD FORK - Checkout Medusa Crawler gem instead "medusa-crawler"

brutuscat

Manage the robots.txt from the Kirby config file

bnomei

Privacy Web Search Engine (not meta, own crawler)

liameno

A pure-Python robots.txt parser with support for modern conventions.

scrapy

A webpack plugin to generate a robots.txt file

itgalaxy

A "robots.txt" parsing and querying library for .NET

TurnerSoftware

Go robots.txt parser

samclarke