ripple icon indicating copy to clipboard operation
ripple copied to clipboard
verified publisher icon joshsh

52 - make use of robots.txt

Open joshsh opened this issue 14 years ago • 0 comments 

LinkedDataSail should request, parse, and comply with the robots.txt of domains which supply one.
Original link: http://code.google.com/p/ripple/issues/detail?id=52

joshsh avatar Apr 08 '11 05:04 joshsh