Bot not detected - WhatsApp

[jLynx]

The bot that is not being detected is:

WhatsApp/2.12.15/i

[sgiehl]

Not sure if that should be detected as bot. Whatsapp is an mobile app and it should already be detected as those. I guess the useragent is used when getting page content for previewing links within a chat. So I guess it's more or less triggered by the user and no "automatic" bot

[jLynx]

As much as it is triggered by the user, it is still a bit as when a user clicks a link, it has the whatsapp bot still visit the Page and then the user visits it, I guess it just collects meta data for its database. But I would still call it a bot since the initial view it's getting isn't the real user

[thE-iNviNciblE]

Would be nice to add this "bot"

[ghost]

Indeed it is a bot.

When you type (or paste) a URL within a WhatsApp conversation, the WhatsApp server loads the URL and parses the Open Graph meta tags. More specifically, it loads the og:image meta tag, which usually loads a relevant image of the URL.

It is common to see two hits, like this:

"GET / HTTP/1.1" 206 11133 "-" "WhatsApp/2.16.16/i"
"GET /images/logo.png HTTP/1.1" 206 18093 "-" "WhatsApp/2.16.16/i"

The first loads the URL as it was typed or pasted in the WhatsApp conversation, the second loads the relevant og:image meta tag, in this case a logo.

[whimsicaldreamer]

Is there any fix to it yet?

[jLynx]

Any update to this issue?

[Ethreal]

👍 Any updates? It's a snippet request and it's straightforward to treat it as a ‘bot’

[Findus23]

I have now tested this myself and indeed during writing of a URL, whatsapp fetches the page to show the preview. (even before sending the message).

But the IP address isn't the one of the Whatsapp servers, but the one of the phone, so the app itself fetches the meta tags.

198.51.100.0 - - [13/Sep/2019:17:24:29 +0200] "GET / HTTP/1.1" 200 1036 "-" "WhatsApp/2.19.229 A"
198.51.100.0 - - [13/Sep/2019:17:24:32 +0200] "GET / HTTP/1.1" 200 1036 "-" "WhatsApp/2.19.229 A"
198.51.100.0 - - [13/Sep/2019:17:24:32 +0200] "GET /te HTTP/1.1" 200 1036 "-" "WhatsApp/2.19.229 A"
198.51.100.0 - - [13/Sep/2019:17:24:33 +0200] "GET /ted HTTP/1.1" 200 1036 "-" "WhatsApp/2.19.229 A"
198.51.100.0 - - [13/Sep/2019:17:24:33 +0200] "GET /tedt HTTP/1.1" 200 1036 "-" "WhatsApp/2.19.229 A"
198.51.100.0 - - [13/Sep/2019:17:24:34 +0200] "GET /tes HTTP/1.1" 200 1036 "-" "WhatsApp/2.19.229 A"
198.51.100.0 - - [13/Sep/2019:17:24:35 +0200] "GET /test HTTP/1.1" 200 1036 "-" "WhatsApp/2.19.229 A"
198.51.100.0 - - [13/Sep/2019:17:24:53 +0200] "GET /t HTTP/1.1" 200 1036 "-" "WhatsApp/2.19.229 A"
198.51.100.0 - - [13/Sep/2019:17:24:53 +0200] "GET /hg HTTP/1.1" 200 1036 "-" "WhatsApp/2.19.229 A"
198.51.100.0 - - [13/Sep/2019:17:24:53 +0200] "GET /hgf HTTP/1.1" 200 1036 "-" "WhatsApp/2.19.229 A"
198.51.100.0 - - [13/Sep/2019:17:24:54 +0200] "GET /hgff HTTP/1.1" 200 1036 "-" "WhatsApp/2.19.229 A"
198.51.100.0 - - [13/Sep/2019:17:24:54 +0200] "GET /hgfff HTTP/1.1" 200 1036 "-" "WhatsApp/2.19.229 A"
198.51.100.0 - - [13/Sep/2019:17:24:54 +0200] "GET /hgffff HTTP/1.1" 200 1036 "-" "WhatsApp/2.19.229 A"
198.51.100.0 - - [13/Sep/2019:17:24:54 +0200] "GET /hgfffff HTTP/1.1" 200 1036 "-" "WhatsApp/2.19.229 A"
198.51.100.0 - - [13/Sep/2019:17:24:54 +0200] "GET /hgffffff HTTP/1.1" 200 1036 "-" "WhatsApp/2.19.229 A"
198.51.100.0 - - [13/Sep/2019:17:24:54 +0200] "GET /hgfffffff HTTP/1.1" 200 1036 "-" "WhatsApp/2.19.229 A"
198.51.100.0 - - [13/Sep/2019:17:24:55 +0200] "GET /hgffffffff HTTP/1.1" 200 1036 "-" "WhatsApp/2.19.229 A"
198.51.100.0 - - [13/Sep/2019:17:25:05 +0200] "GET / HTTP/1.1" 200 1036 "-" "WhatsApp/2.19.229 A"

[thE-iNviNciblE]

it seems not to be added, why?

you could count this as social media traffic....

[sgiehl]

it should be detected as mobile app. See https://github.com/matomo-org/device-detector/blob/master/regexes/client/mobile_apps.yml#L82-L85

[thE-iNviNciblE]

if i test this with web.whatsapp.com and add a link from the shop, i cant see this request. maybe the snippet generator doesn't open the page with javascript and can't be tracked.

Maybe the WhatsApp "Bot" is using grabing the source of html/text. I can see the call in my access.log from the webserver.

[liviuconcioiu]

This issue should be closed.

[Findus23]

Just to make this issue clear, @liviuconcioiu, why do you think this should be closed?

One could argue if the app on the phone should count as a bot, but as it is making automated requests the user doesn't initialize, I think excluding it and detecting it as a bot instead of an app wouldn't be unreasonable.

[sanchezzzhak]

if we consider useragent ^WhatsApp/\d+([\d+\.]+) A$ to be a bot, we may break the page preview functionality in the WhatsApp app. Since some pages do not show the content to the bot.

Similar functionality is available in applications SkypePreview, Telegram.

If Skype SkypePreview is a bot, then this user agent should also be considered a bot, based on the precedent

- 
  user_agent: Mozilla/5.0 (Windows NT 6.1; WOW64) SkypeUriPreview Preview/0.5
  bot:
    name: Skype URI Preview
    category: Service Agent
    url: ""
    producer:
      name: Skype Communications S.à.r.l.
      url: https://www.skype.com