Crawl speed limit and obeying crawl-delay?

[giorgio79]

Hello,

I would like to set an x second crawl delay for sites to avoid hammering the crawled site. So, access the next page on a domain every x second.

In addition, if robots.txt declares a crawl delay, then override the user delay setting with the one in robots.txt. http://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl-delay_directive?