apache-ultimate-bad-bot-blocker icon indicating copy to clipboard operation
apache-ultimate-bad-bot-blocker copied to clipboard

[ADD] Bytedance hidden bot

Open bohwaz opened this issue 10 months ago • 1 comments

Is this an Addition / Removal Request?

Addition

Please List the User-Agent string or Referrer to be added/removed

BrowserMatchNoCase "(?:\b)Build/MRA58N(?:\b)" badbot
BrowserMatchNoCase "(?:\b)Build/OPD3.170816.012(?:\b)" badbot
BrowserMatchNoCase "(?:\b)Build/LRX21T(?:\b)" badbot
BrowserMatchNoCase "(?:\b)CPU iPhone OS 11_0 like Mac OS X.*Chrome/(?:\b)" badbot

Please explain why it should be added

These are parts of user-agents that are used by Bytedance, coming from users of their apps (ie. TikTok). Bytedance are trying to hide their traffic, but these user agents are very weird, for example, iOS 11 running Chrome, or old Android devices.

For Additions: Please include a log sample 3-5 lines is adequate

XXXX:443 74.221.151.32 - - [09/Jan/2025:00:01:39 +0100] "GET /xxx/doc/xxxx/www/admin/.htaccess HTTP/1.1" 200 3188 "-" "Mozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.6530.1545 Mobile Safari/537.36"
XXXX:443 75.229.229.57 - - [09/Jan/2025:00:01:39 +0100] "GET /xxx/draft1/tree?ci=yyyy&name=src%2Ftemplates%2Fconfig%2Fcatxxxx&type=tree HTTP/1.1" 200 6970 "-" "Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.9180.1767 Mobile Safari/537.36"
XXXX:443 75.180.26.109 - - [09/Jan/2025:00:01:40 +0100] "GET /xxx/draft1/finfo?ci=yyyy&name=doc%2Findex.md HTTP/1.1" 200 21074 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.9415.1534 Mobile Safari/537.36"
XXXX:443 68.12.64.235 - - [09/Jan/2025:00:01:40 +0100] "GET /xxx/draft1/finfo?ci=merge-in%3A59f69df&name=doc%2Fadmin%2Fbxxxs.md HTTP/1.1" 200 18713 "-" "Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2451.1463 Mobile Safari/537.36"

bohwaz avatar Jan 09 '25 04:01 bohwaz

Other sources on this: https://www.webmasterworld.com/search_engine_spiders/5088284.htm https://xenforo.com/community/threads/known-bots.148723/page-4

Webmaster World
News and discussion for the independent webprofessional
XenForo community
The lists can be used to block by user agent at the web server. Those two places are just good sources of user agent info.

Yeah I just... Pictured in my mind a hundred guys copying the .txt files and adding them to their robots.txt file, thinking that'll work.

bohwaz avatar Jan 09 '25 04:01 bohwaz