Need warning for robots.txt

[WilliamKF]

It would be nice if when finding a robots.txt file that excludes recursion and in verbose mode states:

    Info       Access denied by robots.txt, checked only syntax.

That a prominent warning be given, especially if that was the root node visited. Otherwise, when not in verbose mode, nothing happens and it is not clear why.

[mkormendy]

How do we ignore robots.txt?

[dpalic]

Thank you for the issue report. Sadly this project is dead, and a new team is around with https://github.com/linkcheck/linkchecker for more details please see: #708 Also please close this issue and report it freshly on the new repo https://github.com/linkcheck/linkchecker/issues if your issue still persists

[mkormendy]

Seems to be the same repo. Check your link.

[dpalic]

please recheck your check. The links are fine... https://github.com/wummel/linkchecker (old/dead) https://github.com/linkcheck/linkchecker (new/alive)