itsy icon indicating copy to clipboard operation
itsy copied to clipboard
verified publisher icon dakrone

Obeying robots.txt?

Open giorgio79 opened this issue 11 years ago • 3 comments

Any plans for robots.txt compliance?

giorgio79 avatar May 24 '14 14:05 giorgio79

Itsy already looks at robots.txt, see: https://github.com/dakrone/itsy/blob/master/src/itsy/core.clj#L133

dakrone avatar May 24 '14 16:05 dakrone

Nice. Do you plan on implementing crawl-delay too? http://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl-delay_directive Or just a general delay setting?

giorgio79 avatar May 24 '14 16:05 giorgio79

I think supporting the crawl-delay parameter would be the best, I'll add it to the todo.

dakrone avatar May 30 '14 11:05 dakrone