Add support for Crawl-Delay from robots.txt

[pypt]

https://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl-delay_directive

[gavishpoddar]

Hi @pypt is the issue still open, If yes I would love to fix it

[gavishpoddar]

Issue #4 as well if it's still open