proofofconcept icon indicating copy to clipboard operation
proofofconcept copied to clipboard

same robots.txt for two domains

Open bhpayne opened this issue 1 year ago • 2 comments

Currently both https://derivationmap.net/robots.txt and https://allofphysics.com/robots.txt both resolve to the same file which contains

User-agent: *
Allow: /

Sitemap: https://derivationmap.net/sitemap.txt

I may need to have two separate robots.txt files for the different domains? The other robots.txt would contain

User-agent: *
Allow: /

Sitemap: https://allofphysics.com/sitemap.txt

but then I'd need to also have a separate sitemap.txt that references <allofphysics.com> as the domain.

bhpayne avatar Dec 28 '23 11:12 bhpayne

Two different nginx approaches:

https://www.digitalocean.com/community/questions/nginx-same-root-folder-for-multiple-website-different-robots-txt

https://stackoverflow.com/questions/26308779/nginx-different-robots-txt-for-alternate-domain

bhpayne avatar Dec 28 '23 11:12 bhpayne

For now I'm going to hold off on fixing what might not actually be an issue.

If the Google index doesn't include allofphysics.com then a separate robots.txt might be relevant.

bhpayne avatar Dec 28 '23 11:12 bhpayne