vue-telescope-analyzer icon indicating copy to clipboard operation
vue-telescope-analyzer copied to clipboard

Published 20 hours ago • verified publisher icon nuxtlabs

Respect the directives in robots.txt

Open bdrtsky opened this issue 3 years ago • 0 comments

Add Vue Telescope Bot user-agent and respect robots.txt. Also add instructions in README.

Here's how Google explains it:

https://developers.google.com/search/docs/advanced/crawling/googlebot https://developers.google.com/search/docs/advanced/crawling/control-what-you-share?visit_id=637475245791824176-1791820022&rd=2

bdrtsky avatar Jan 29 '21 13:01 bdrtsky