Avaiga
Update robots.txt to Index Only Latest Version of Documentation #1960
Update robots.txt to Index Only Latest Version of Documentation
PR Description: This PR updates the robots.txt file in the Taipy documentation repository to ensure that only the latest version of the documentation is indexed by search engines. The changes made include:
-
Allowing search engines to crawl and index the /en/latest/ version of the documentation.
-
Disallowing search engines from accessing any other /en/ versions (e.g., /en/release-3.0/) to prevent outdated versions from appearing in search results.
-
Adding a Sitemap reference (https://docs.taipy.io/sitemap.xml) to help search engines efficiently crawl the allowed URLs.
These changes are aimed at improving SEO by ensuring that users are directed to the most up-to-date version of the documentation when searching online.
Let me know if you'd like any further adjustments or additional details!
