More output on errors with external links on debug log level

[r4fterman]

I am currently running into 403 and ExternalTimeout errors for some links on the internet I am having trouble to identify the root cause. Using curl for the same links is working fine.

There is the option to set log-level to debug but this is not much of a help in indentifying the root of the failing check. Output is only.

E.g.: Status 403

[0m[35m  fresh --- mysql-jdbc.html --> https://dev.mysql.com/downloads/connector/j/
[0m[34m  hitting --- mysql-jdbc.html --> https://dev.mysql.com/downloads/connector/j/
[0m[31m  Non-OK status: 403 --- mysql-jdbc.html --> https://dev.mysql.com/downloads/connector/j/

Timeout

[0m[35m  fresh --- custom.html --> https://www.dnb.com/de-de/produkte-services/dun-bradstreet/dnb-duns-nummer
[0m[34m  hitting --- custom.html --> https://www.dnb.com/de-de/produkte-services/dun-bradstreet/dnb-duns-nummer
[0m[31m  request exceeded our ExternalTimeout --- custom.html --> https://www.dnb.com/de-de/produkte-services/dun-bradstreet/dnb-duns-nummer

On DEBUG level I expect the following Information to be added:

• Final URL that is crawled (there might some adaptions to the URL as seen. in the code)
• Request method is used (from code it seems to be GET)
• Response code
• Response content