apache-ultimate-bad-bot-blocker
apache-ultimate-bad-bot-blocker copied to clipboard
[ADD] Bytedance hidden bot
Is this an Addition / Removal Request?
Addition
Please List the User-Agent string or Referrer to be added/removed
BrowserMatchNoCase "(?:\b)Build/MRA58N(?:\b)" badbot
BrowserMatchNoCase "(?:\b)Build/OPD3.170816.012(?:\b)" badbot
BrowserMatchNoCase "(?:\b)Build/LRX21T(?:\b)" badbot
BrowserMatchNoCase "(?:\b)CPU iPhone OS 11_0 like Mac OS X.*Chrome/(?:\b)" badbot
Please explain why it should be added
These are parts of user-agents that are used by Bytedance, coming from users of their apps (ie. TikTok). Bytedance are trying to hide their traffic, but these user agents are very weird, for example, iOS 11 running Chrome, or old Android devices.
For Additions: Please include a log sample 3-5 lines is adequate
XXXX:443 74.221.151.32 - - [09/Jan/2025:00:01:39 +0100] "GET /xxx/doc/xxxx/www/admin/.htaccess HTTP/1.1" 200 3188 "-" "Mozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.6530.1545 Mobile Safari/537.36"
XXXX:443 75.229.229.57 - - [09/Jan/2025:00:01:39 +0100] "GET /xxx/draft1/tree?ci=yyyy&name=src%2Ftemplates%2Fconfig%2Fcatxxxx&type=tree HTTP/1.1" 200 6970 "-" "Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.9180.1767 Mobile Safari/537.36"
XXXX:443 75.180.26.109 - - [09/Jan/2025:00:01:40 +0100] "GET /xxx/draft1/finfo?ci=yyyy&name=doc%2Findex.md HTTP/1.1" 200 21074 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.9415.1534 Mobile Safari/537.36"
XXXX:443 68.12.64.235 - - [09/Jan/2025:00:01:40 +0100] "GET /xxx/draft1/finfo?ci=merge-in%3A59f69df&name=doc%2Fadmin%2Fbxxxs.md HTTP/1.1" 200 18713 "-" "Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2451.1463 Mobile Safari/537.36"
Other sources on this: https://www.webmasterworld.com/search_engine_spiders/5088284.htm https://xenforo.com/community/threads/known-bots.148723/page-4
Webmaster WorldNews and discussion for the independent webprofessional
XenForo community
The lists can be used to block by user agent at the web server. Those two places are just good sources of user agent info.Yeah I just... Pictured in my mind a hundred guys copying the .txt files and adding them to their robots.txt file, thinking that'll work.