Block AI bots and crawlers via robots.txt

[flodolo]

Should we block LLM bots via robots.txt? I couldn't help noticing the amount of requests coming from Claude for non-existing strings when checking the logs for neterror.dtd.

This is the list I came up with recently for Transvision

User-agent: AhrefsBot
User-agent: AliyunSecBot
User-agent: Amazonbot
User-agent: Barkrowler
User-agent: BLEXBot
User-agent: Bytespider
User-agent: GPTBot
User-agent: meta-externalagent
User-agent: MJ12bot
User-agent: PetalBot
User-agent: SemrushBot
Disallow: /

Possible useful to add based on the log

User-agent: ClaudeBot