ol-infrastructure
ol-infrastructure copied to clipboard
robots.txt files for draft.ocw.mit.edu, live-qa and draft-qa servers
User Story
Users don't want to get search results from google for live-qa.ocw.mit.edu or any of the other qa and draft servers
Acceptance Criteria
- [ ] Don't break the robots.txt file for https://ocw.mit.edu
- [ ] Write a github action to test the contents of https://ocw.mit.edu/robots.txt -- it should always include
Allow: /
andSitemap: https://ocw.mit.edu/sitemap.xml
- [ ] https://live-qa.ocw.mit.edu/robots.txt should be set to deny all
- [ ] https://draft-qa.ocw.mit.edu/robots.txt should be set to deny all
- [ ] https://draft.ocw.mit.edu/robots.txt should be set to deny all (currently it returns a 503 error)