ol-infrastructure icon indicating copy to clipboard operation
ol-infrastructure copied to clipboard

robots.txt files for draft.ocw.mit.edu, live-qa and draft-qa servers

Open pdpinch opened this issue 2 years ago • 7 comments

User Story

Users don't want to get search results from google for live-qa.ocw.mit.edu or any of the other qa and draft servers

Acceptance Criteria

  • [ ] Don't break the robots.txt file for https://ocw.mit.edu
  • [ ] Write a github action to test the contents of https://ocw.mit.edu/robots.txt -- it should always include Allow: / and Sitemap: https://ocw.mit.edu/sitemap.xml
  • [ ] https://live-qa.ocw.mit.edu/robots.txt should be set to deny all
  • [ ] https://draft-qa.ocw.mit.edu/robots.txt should be set to deny all
  • [ ] https://draft.ocw.mit.edu/robots.txt should be set to deny all (currently it returns a 503 error)

pdpinch avatar Aug 05 '22 02:08 pdpinch