helix-home
helix-home copied to clipboard
Make robots.txt part of the boilerplate
Is your feature request related to a problem? Please describe.
When creating a new website all pages under .page
and .live
have default robots.txt
which is great because you don't want to have your development website indexed. But at the moment you create your BYOCDN integration, the default robots.txt
is not served anymore if you have not added one until that point. This can have side effects where the website you are working on start being indexed by search engines.
Steps to reproduce:
- Start a new website
- Configure your CDN (example.com)
- Now on https://example.com you have no
robots.txt
Describe the solution you'd like
- Have a
robots.txt
in the boilerplate with the same content - Add
Modifying the robots.txt
as part of the launch checks