crystal-libraries-needed icon indicating copy to clipboard operation
crystal-libraries-needed copied to clipboard

Published 20 hours ago • verified publisher icon crystal-community

A port of, or comparable functionality too Ruby Robotstxt

Open johnjansen opened this issue 7 years ago • 1 comments

see https://github.com/gjtorikian/robotstxt-parser

johnjansen avatar Jul 13 '17 19:07 johnjansen

see

https://yoast.com/ultimate-guide-robots-txt/ https://en.wikipedia.org/wiki/Robots_exclusion_standard

separately consider sitemaps https://en.wikipedia.org/wiki/Sitemaps

johnjansen avatar Jul 13 '17 19:07 johnjansen