Crawlers should have an option to respect robots.txt

[jakubbalada]

trafficstars

Now you have to write your own function to parse and respect target website's robots.txt file. Common function in an SDK (utils.js probably) for that would be great.

[LeMoussel]

I propose to use Robots Parser library with this common functions in utils.js :

• getRobotsTxt(url)
• isAllowedRobotsTxt(url, ua)
• isDisallowedRobotsTxt(url, ua)
• getCrawlDelayRobotsTxt(ua)

[mgifford]

Ok, so what is the status of this? It's not clear where this has been addressed (or why it hasn't yet).

[mnmkng]

It's not implemented. There were not that many users requesting it and it's easy enough to implement by the users who need it. We might add it in the future, but there's no timeline.

[mgifford]

Is this documented somewhere? I'm not interested in handcrafting one per URL, but rather looking at the robots.txt file and extracting the information about what to skip to be respectful of the site owner's direction.

[mnmkng]

See the comment above by LeMoussel.