/keystatic route can be not indexed

[solrik]

Expected Behaviour

website_url/keystatic should not be listed in search engine.

Actual Behaviour

website_url/keystaic now can be crawled and indexed by search engine.

Additional Context

• When users creates a site, sometimes the unwanted routes are not added to robots.txt, and once crawled, google will index it even if the route is added in robots.txt later. To solve it, no-index could be added to keystatic route by default.
• Users don't have to add it to the robots.txt manually.
• Might avoid bots crawling in future.

Related reading

Google Dev Doc

[JPrisk]

great suggestion @solrik, we should include this in all the starter templates 🙌

[t1gu1]

You can add that in your robots.txt (Should help)

User-agent: *
Allow: /
Disallow: /keystatic/