dweb-gateway icon indicating copy to clipboard operation
dweb-gateway copied to clipboard
verified publisher icon internetarchive

Handle robots.txt

Open mitra42 opened this issue 7 years ago • 0 comments

Seeing queries to /arc/archive.org/robots.txt which probably means they are going to dweb.archive.org/robots.txt

  • Decide on policy - probably block it.
  • Implement on dweb.archive.org and probably on dweb.me/arc/archive.org/robots.txt and dweb.me/robots.txt

mitra42 avatar May 31 '18 19:05 mitra42