cisagov
Update robots.txt
Updated robots.txt so well-behaved search engine crawlers will not index Crossfeed instances, as I assume they're not meant to appear in search engine results due to the potentially sensitive information they collect.
🗣 Description
💭 Motivation and context
🧪 Testing
✅ Pre-approval checklist
- [x] This PR has an informative and human-readable title.
- [x] Changes are limited to a single goal - eschew scope creep!
- [ ] All future TODOs are captured in issues, which are referenced in code comments.
- [ ] All relevant type-of-change labels have been added.
- [ ] I have read the CONTRIBUTING document.
- [x] These code changes follow cisagov code standards.
- [x] All relevant repo and/or project documentation has been updated to reflect the changes in this PR.
- [x] Tests have been added and/or modified to cover the changes in this PR.
- [ ] All new and existing tests pass.
✅ Pre-merge checklist
- [ ] Revert dependencies to default branches.
- [ ] Finalize version.
✅ Post-merge checklist
- [ ] Create a release.
@cqueern Thanks for taking an interest in Crossfeed! You bring up an interesting point about whether or not we want to be indexed. I believe that allowing indexing was a deliberate decision, but I will look into it further.
My pleasure! It's an important project.
Sounds good. If the decision is to exclude all robots from the entire server, suggest that line 3 in the robots.txt file remain
Disallow: /
If the decision is to allow all robots complete access, suggest that line 3 in the robots.txt file be updated to
Disallow: