[Request] Add robots.txt parsing

[joshua-bn]

Would be nice to have the ability to parse robots.txt like RSS feeds. $web->robots

https://github.com/bopoda/robots-txt-parser is a library. Not sure if it is the one to use here but it seems to do the job