crawlee-python icon indicating copy to clipboard operation
crawlee-python copied to clipboard
verified publisher icon apify

add `Sitemap` parser utility

Open Mantisus opened this issue 8 months ago • 1 comments

Sitemap parser is closely related to RobotsTxtFile. But it is not used for support `respect_robots_txt_file'

  • Related: #1144

Mantisus avatar Apr 17 '25 23:04 Mantisus

This should also include a SitemapRequestLoader implementation so that you can easily plug a sitemap into your crawler.

janbuchar avatar Apr 22 '25 09:04 janbuchar