proofofconcept icon indicating copy to clipboard operation
proofofconcept copied to clipboard

same robots.txt for two domains

Open bhpayne opened this issue 6 months ago • 2 comments

Currently both https://derivationmap.net/robots.txt and https://allofphysics.com/robots.txt both resolve to the same file which contains

User-agent: *
Allow: /

Sitemap: https://derivationmap.net/sitemap.txt

I may need to have two separate robots.txt files for the different domains? The other robots.txt would contain

User-agent: *
Allow: /

Sitemap: https://allofphysics.com/sitemap.txt

but then I'd need to also have a separate sitemap.txt that references <allofphysics.com> as the domain.

bhpayne avatar Dec 28 '23 11:12 bhpayne