Build in recognition of search bot user agents

[davideanderson]

Feature Suggestion: It seems that it would be relatively common use case to desire to know if the agent is a search bot, so building in the user agents for common search bots seems like a reasonable feature.

Google: https://support.google.com/webmasters/answer/1061943?hl=en Bing: https://www.bing.com/webmaster/help/which-crawlers-does-bing-use-8c184ec0

[lancedikson]

It's ok. Could you prepare a PR?

[everget]

@lancedikson hope this helps

Majestic-12 Web Crawler:

/mj12bot|majestic12/i

Google bots (matches the all from https://support.google.com/webmasters/answer/1061943?hl=en):

/(mediapartners|apis|adsbot)?[ -]?\(?google(bot)?/i

Bing bots (mathes the all from https://www.bing.com/webmaster/help/which-crawlers-does-bing-use-8c184ec0):

/msn|adidx|bing(bot|preview)/i

Chinese search engine Baidu crawlers:

/baiduspider/i

Chinese search engine Sogou crawlers:

/sogou/i

[daniel-nagy]

DuckDuckGo

Mozilla/5.0 (compatible; DuckDuckBot-Https/1.1; https://duckduckgo.com/duckduckbot)

[gurodrigues]

I found this list with all user agent crawlers

http://www.user-agents.org/