ultimate-sitemap-parser icon indicating copy to clipboard operation
ultimate-sitemap-parser copied to clipboard

Published 20 hours ago • verified publisher icon mediacloud

Add support for Crawl-Delay from robots.txt

Open pypt opened this issue 6 years ago • 2 comments

https://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl-delay_directive

pypt avatar Nov 27 '18 10:11 pypt

Hi @pypt is the issue still open, If yes I would love to fix it

gavishpoddar avatar Mar 31 '21 16:03 gavishpoddar

Issue #4 as well if it's still open

gavishpoddar avatar Mar 31 '21 16:03 gavishpoddar