crawlerdetect
crawlerdetect copied to clipboard
Please add SpierLing to bots list
https://www.sketchengine.eu/ scrape data from the web, repackage, and sell it. We'd like to be able to reject their crawler. Below are the details from them about their user agent:
User-Agent: Mozilla/5.0 (compatible; SpiderLing (a SPIDER for LINGustic research)
# We observer robots.txt. To block SpiderLing from downloading your website, use
User-agent: SpiderLing
Disallow: /
And some tweets about it: https://twitter.com/SketchEngine/status/1368861556906356742?s=20
Mahalo 🤙🏽