helix-home icon indicating copy to clipboard operation
helix-home copied to clipboard

Make robots.txt part of the boilerplate

Open fe-lix- opened this issue 1 year ago • 0 comments

Is your feature request related to a problem? Please describe. When creating a new website all pages under .page and .live have default robots.txt which is great because you don't want to have your development website indexed. But at the moment you create your BYOCDN integration, the default robots.txt is not served anymore if you have not added one until that point. This can have side effects where the website you are working on start being indexed by search engines.

Steps to reproduce:

  1. Start a new website
  2. Configure your CDN (example.com)
  3. Now on https://example.com you have no robots.txt

Describe the solution you'd like

  1. Have a robots.txt in the boilerplate with the same content
  2. Add Modifying the robots.txt as part of the launch checks

fe-lix- avatar Jul 05 '23 14:07 fe-lix-