robots.txt icon indicating copy to clipboard operation
robots.txt copied to clipboard

Published 20 hours ago • verified publisher icon fooock

Implement robots.txt download priority system

Open fooock opened this issue 5 years ago • 0 comments

Differentiate URLs sent from the crawl API and others sent by users (when its query is not found). In this case, the priority is to download the robots file of the URL sent by the user.

We can have multiple levels of priority to allow the development of more use cases and/or business requirements.

fooock avatar Jun 30 '19 14:06 fooock