proofofconcept
proofofconcept copied to clipboard
same robots.txt for two domains
Currently both https://derivationmap.net/robots.txt and https://allofphysics.com/robots.txt both resolve to the same file which contains
User-agent: *
Allow: /
Sitemap: https://derivationmap.net/sitemap.txt
I may need to have two separate robots.txt files for the different domains? The other robots.txt would contain
User-agent: *
Allow: /
Sitemap: https://allofphysics.com/sitemap.txt
but then I'd need to also have a separate sitemap.txt that references <allofphysics.com> as the domain.