crawlerdetect icon indicating copy to clipboard operation
crawlerdetect copied to clipboard

Please add SpierLing to bots list

Open kmahelona opened this issue 3 years ago • 0 comments

https://www.sketchengine.eu/ scrape data from the web, repackage, and sell it. We'd like to be able to reject their crawler. Below are the details from them about their user agent:

User-Agent: Mozilla/5.0 (compatible; SpiderLing (a SPIDER for LINGustic research)
# We observer robots.txt. To block SpiderLing from downloading your website, use 
User-agent: SpiderLing
Disallow: /

And some tweets about it: https://twitter.com/SketchEngine/status/1368861556906356742?s=20

Mahalo 🤙🏽

kmahelona avatar Mar 17 '21 21:03 kmahelona