crawl icon indicating copy to clipboard operation
crawl copied to clipboard
verified publisher icon benjaminestes

Sitemap crawling should respect robots.txt

Open benjaminestes opened this issue 6 years ago • 0 comments

If for some reason a site blocks its own sitemap with a robots.txt file, the crawler should respect that and not request the sitemaps in sitemap mode.

benjaminestes avatar Apr 29 '19 18:04 benjaminestes