/favicon.ico should be detected before added to the url list

[swtrse]

File: URLDetector.php

The current way if always add /favicon.ico generates unessesary 404 errors. In most WP sites favicon is now set in admin panel (over Design-Customzier or a Plugin). The favicon is placed in the media library and an html meta tag is inserted on each page that references the favicon. There is no need to rely on an favicon.ico on site root anymore.

Since favicon is not present on site root, crawling will throw an 404 message. The presence of favicon on root level should be tested and only if present it should be added in the url list.

Same goes for robots.txt and sitemap.xml