Option to specify bot for robots.txt

[Garrett-R]

For a site that has a robots.txt like:

User-agent: *
Disallow: /

User-agent: Googlebot
Allow: /
Disallow: /login

it would be nice if there was an option to run the script as Googlebot, so it follow those rules.

[c4software]

It's a really nice idea !

I will look at this soon.

[Garrett-R]

Cool! My team can also submit a PR if that helps!