firecrawl
firecrawl copied to clipboard
[Feat] Dedup duplicate links in sitemap
Customer problem:
"When I crawl websites with firecrawl I'll sometimes get essentially the same links eg https://site.com/, https://www.site.com/, https://www.site.com/, when in reality they're all the same page"
It would be good to dedup these