Crawler-Detect icon indicating copy to clipboard operation
Crawler-Detect copied to clipboard

Agent with '+http...' almost always a bot.

Open bartonlp opened this issue 11 months ago • 1 comments

Any time you find a user agent with '+http' it is almost always a bot. So for example, '+https://help.something.com' will probably be a bot. The 'good' bots almost always have a place where you can see who they are.

bartonlp avatar Jul 26 '23 22:07 bartonlp

"Probably" is a bit loose for my liking, unless you can convince me otherwise 🤔

JayBizzle avatar Aug 08 '23 18:08 JayBizzle