gophervista icon indicating copy to clipboard operation
gophervista copied to clipboard

Published 20 hours ago • verified publisher icon benjojo

robots.txt

Open jamestomasino opened this issue 5 years ago • 2 comments

Just a note that if you are publishing your crawler results to please respect sites which host a robots.txt restricting spidering.

For instance: gopher://gopher.black/0/robots.txt

Thanks

jamestomasino avatar Aug 12 '18 18:08 jamestomasino