Racheet Dave

I've had to do the same patch, and I agree that it's something that is definitely needed. The codebase shouldn't assume that all websites have a robots.txt when the accepted...