python-sitemap
python-sitemap copied to clipboard
Option to specify bot for robots.txt
For a site that has a robots.txt like:
User-agent: *
Disallow: /
User-agent: Googlebot
Allow: /
Disallow: /login
it would be nice if there was an option to run the script as Googlebot, so it follow those rules.
It's a really nice idea !
I will look at this soon.
Cool! My team can also submit a PR if that helps!