Better de-duplication of URLs

[croqaz]

When discovering URLs from different seeds, the URLs are not deduplicated if they are found in multiple seeds. There is local de-duplication during discovery, and there's also the built-in DupeFilters. Would need investigation.