smart-url-fuzzer icon indicating copy to clipboard operation
smart-url-fuzzer copied to clipboard

Published 20 hours ago • verified publisher icon avilum

Scan robots.txt

Open avilum opened this issue 6 years ago • 1 comments

Make a set of all the endpoints in robots.txt, and add them to the active endpoints.

avilum avatar Aug 21 '18 11:08 avilum

Automatically check if the robots.txt file of the domain contains any word in the words list.

avilum avatar Dec 01 '20 11:12 avilum