Crawler-Detect
Crawler-Detect copied to clipboard
Agent with '+http...' almost always a bot.
Any time you find a user agent with '+http' it is almost always a bot. So for example, '+https://help.something.com' will probably be a bot. The 'good' bots almost always have a place where you can see who they are.
"Probably" is a bit loose for my liking, unless you can convince me otherwise 🤔