preact-www
preact-www copied to clipboard
chore: Add `noindex` meta tag to v8 doc pages
Closes #910
Re: v8 docs being indexed (often higher than v10).
The meta tag is the correct course, rather than say Disallow
in the robots.txt
, as Disallow
only blocks the crawler from discovering the page while navigating our site. While this often does have an effect, external links to the v8 docs could keep them ranking highly, which would be an issue. What we actually want to do is signal that these pages shouldn't be indexed, not hide it from discovery, hence, the meta tag.
https://developers.google.com/search/docs/crawling-indexing/robots/robots-faq?visit_id=637994089979808237-3634207106&rd=1#h17
~~What I am unsure of is if we need to do client-side addition/removal of this tag upon navigation. I haven't quite been able to get a definitive answer on that, does anyone know?~~ Edit: Upon doing some more reading I don't think that will be an issue.