firecrawl icon indicating copy to clipboard operation
firecrawl copied to clipboard

[Feat] Dedup duplicate links in sitemap

Open calebpeffer opened this issue 6 months ago • 0 comments

Customer problem:

"When I crawl websites with firecrawl I'll sometimes get essentially the same links eg https://site.com/, https://www.site.com/, https://www.site.com/, when in reality they're all the same page"

It would be good to dedup these

calebpeffer avatar Aug 21 '24 17:08 calebpeffer