YarnSeemannsgarn issues

Repositories
Issues
Comments

Results 2 issues of


                                            YarnSeemannsgarn

Fix import for reader

[IMP]: Respectation of robots.txt

I found an example [https://booking.com/robots.txt](https://booking.com/robots.txt) where sitemaps are marked as **Disallowed** ``` Sitemap: https://www.booking.com/sitembk-index-https.xml` User-agent: Baiduspider Disallow: /sitembk-index-https.xml ``` I suggest to add an option **respectRobotsTxt** to the parser which...