RobotsDisallowed
RobotsDisallowed copied to clipboard
Published
20 hours ago •
danielmiessler
danielmiessler
A curated list of the most common and most interesting robots.txt disallowed directories.
Results
5
RobotsDisallowed issues
Sort by
recently updated
recently updated
newest added
Trailing slash was causing unwanted redirects instead of a proper response, so, I removed trailing slashes from all items in the list using regex.
I didn't got that even though having Top-N.txt wordlist, there are more in archive folder? what are they? is it same?
Metadata
1.3k
Stars
315
Forks
Watchers
Owner
Metadata
A curated list of the most common and most interesting robots.txt disallowed directories.