Gideon Catz

🐛 [Bug]: Crawling gets confused by urls ending with /sitemap

1

### Archon Version 0.1.0 ### Bug Severity 🟡 Medium - Affects functionality ### Bug Description I was trying to crawl this URL: https://nx.dev/see-also/sitemap That page does have all the links...

Feature request: Support for scraping instructions

2

When scraping for documentation, it would be awesome to be able to provide specific instructions for scraping. For example, from the site https://reactnative.dev/sitemap.xml I want to scrape documentation only for...