creepjs icon indicating copy to clipboard operation
creepjs copied to clipboard

just opening one for my research on bot detection and stuff

Open vis2021t opened this issue 2 years ago • 138 comments

I looked over the tls fingerprinting, You talked about but there is something I read at akamai research where they stated that bot are able to bypass to get on gud side :- https://www.akamai.com/blog/security/bots-tampering-with-tls-to-avoid-detection

I came across a 2 step tls fingerprinting but I lost that pdf 🥲🥲 dammit

Will try to find it but do u know about it?

vis2021t avatar Jul 12 '22 05:07 vis2021t

True, bots can still bypass it. I have some good resources. Have not heard of the 2 step.

  • https://incolumitas.com/pages/TLS-Fingerprint/
  • https://github.com/LeeBrotherston/tls-fingerprinting
    • https://www.youtube.com/watch?v=XX0FRAy2Mec
  • https://github.com/salesforce/ja3 (https://ja3er.com/)
    • https://www.youtube.com/watch?v=oprPu7UIEuk
    • https://developers.cloudflare.com/bots/concepts/ja3-fingerprint/
  • https://scrapfly.io/blog/how-to-avoid-web-scraping-blocking-tls/
  • https://lcamtuf.coredump.cx/p0f3/

abrahamjuliot avatar Jul 12 '22 05:07 abrahamjuliot

True, bots can still bypass it. I have some good resources. Have not heard of the 2 step.

Everything is bypassable in the world of Javascript well Thanks for resources I am looking into them just now

vis2021t avatar Jul 12 '22 06:07 vis2021t

1-s2 0-S0167404821003990-ga1_lrg

I found this chart which maybe something of our interest

vis2021t avatar Jul 12 '22 06:07 vis2021t

I was wondering to look over CVE for specific browser and it's version,

If for demo purpose we can proceed ahead and identify too much info on the device/browser

I know it's actually creepy but comeon it's in the name too lol

It's not a bad idea u know We can identify many things if we play well but I'm not sure it's a gud idea to implement but it's a definitely gud section to look still not sure for implementation.

What do u feel?

  • I think Platform lies should be considered as a part of bot lies, Like We can keep them as I have noticed bots have different level in creepjs bot detection section

vis2021t avatar Jul 13 '22 04:07 vis2021t

Not a bad idea. Maybe start with a test page. What I sometimes do is begin with a test page and experiment/research there. If we get stable results, we can release on the main page. If it has good performance and good fingerprinting, we can implement it in the main fingerprint.

Platform lies part of bot lies

I like this idea. I will look into it.

abrahamjuliot avatar Jul 13 '22 04:07 abrahamjuliot

I am really interested in chrome://chrome-urls/ There are many thing which can make things go really really really deep

++ I am looking over cve which can verify the browser version for us but I was thinking over more of the section of bot detection, hmm and yea I saw there are Many features which are not supported in Chrome android at the section of Chrome flags there is a section for what is not supported on my device maybe can be something of notice? I guess So maybe we can look Into it

vis2021t avatar Jul 13 '22 10:07 vis2021t

This one is interesting… till it gets patched. In Chrome, it can be used to validate if a device is really on macOS.

https://developer.mozilla.org/en-US/docs/Web/API/Web_Share_API#api.navigator.canshare https://bugs.chromium.org/p/chromium/issues/detail?id=1144920

abrahamjuliot avatar Jul 14 '22 02:07 abrahamjuliot

See I told u Cve and bugs are great place for us to look even if it will be patched for later versions it will still be there for people who don't usually update ( I was one of them ) And I know many who don't update

vis2021t avatar Jul 14 '22 04:07 vis2021t

Btw Do u have anything in mind for bot detection ahead?

I mean in the end Creepjs is a bot detection repo sort of itself,

from the section of lies till loosing their expected features

So I was curious if u had something in research lately

Note:- Android and iOs devices never come with Angle as their gpu if they are real, Google emulator Friendly web test had the same thing and I have seen it only in bots till yet when it comes to these 2 os,

It can be a small point

I mean Imagine seeing intel as the gpu of Android device user 😂 aah dude nevermind just want to convey that hardware filter are an essential parts in gpu to

combining confidence methodology it can be a gud charm

vis2021t avatar Jul 14 '22 04:07 vis2021t

I think I will love to go ahead at bugs amd cve section for creepjs Look at this:- 😈

This place is really a treasure for us

Screenshot_20220714-105929_Kiwi Browser

vis2021t avatar Jul 14 '22 05:07 vis2021t

mmm don't u think we should bring up geckodriver too in headless section as Till yet it is focused on chromedriver

vis2021t avatar Jul 14 '22 20:07 vis2021t

Good idea. We should absolutely include geckodriver and more.

abrahamjuliot avatar Jul 16 '22 23:07 abrahamjuliot

bot detection and research

Nothing on my mind, atm. But, ideas are welcome.

gpu hardware filter

This is on my mind. I've been slow to get to it. We should definitely look out for GPU lies in reported mobile devices. Samsung Xclipse 920 has Angle, but I think we can determine Angle is not iOS.

abrahamjuliot avatar Jul 17 '22 00:07 abrahamjuliot

bot detection and research

Nothing on my mind, atm. But, ideas are welcome.

gpu hardware filter

This is on my mind. I've been slow to get to it. We should definitely look out for GPU lies in reported mobile devices. Samsung Xclipse 920 has Angle, but I think we can determine Angle is not iOS.

mm but expect that device almost every device comes with real like mediatek helio or Qualcomm

vis2021t avatar Jul 17 '22 04:07 vis2021t

Hi, was busy with something well let's get back to research

I found something interesting to look at:-

https://github.com/mdn/content/pull/6849

vis2021t avatar Jul 31 '22 10:07 vis2021t

https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=927531

found something to look at

it's regarding 2 step tls fingerprinting

vis2021t avatar Jul 31 '22 10:07 vis2021t

Nice. I wonder if TLS fingerprint is distinct on mobile devices vs desktop. I presume no.

abrahamjuliot avatar Aug 02 '22 04:08 abrahamjuliot

Do u have a report of what is the top 5 browser version Creepjs usually gets to see

I am curious if people use older version as there are bugs and vulnerability if old one is there _ that might be an interesting approach if we go in ethical way

vis2021t avatar Aug 02 '22 05:08 vis2021t

It depends on the date, but the top 5 versions usually consist of versions at or near the latest stable releases of Blink, Gecko, and WebKit. Here's yesterday, for example:

image

We do get a lot of older browsers, though. The window test page contains a pool of browser versions seen in the last 40 days.

I'm sure we would see even older browsers if the code was geared for ES5. Right now, the target is ES2019.

abrahamjuliot avatar Aug 02 '22 23:08 abrahamjuliot

found something

Navigator.connection.type only there for android and ios

can be a part as it is something quite not people hide

if windows and Linux it's not there they says privacy issues........ Like they gave it to android and ios well better for us enj0y

vis2021t avatar Aug 06 '22 22:08 vis2021t

Nice. I plan to add this. Looks like type is only on Android and Chrome OS, but we could use this to determine if a device is really Android/Chrome OS. There are a lot of interesting ways this API can be used for fingerprinting. These are also in client hint headers.

https://wicg.github.io/netinfo/#privacy-considerations https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers#network_client_hints

rtt in Headless Chrome is 0, but I'm not sure if that is always the case and exclusive to headless.

abrahamjuliot avatar Aug 07 '22 00:08 abrahamjuliot

I wanna test the networkinformation type to Google mobile friendly display test

I think majority of the big brand bots uses simulation instead of emulation so it could be a part in terms of bot who are stating to be android but They are not , can be considered as suspicious by us

I am currently learn typescript for js as we are switching at that

I will explore Navigator more deep into every inner parts of it

vis2021t avatar Aug 07 '22 01:08 vis2021t

What is your net speed hope it's not in gbps lol rtt? I checked on my chrome browser ( Kiwi browser [ Android Chromium based browser with pc dev tools ] to look at navigation.connection and research )

here is my result:- Screenshot_20220807-065732_Kiwi Browser

vis2021t avatar Aug 07 '22 01:08 vis2021t

I need to test more in Kiwi. Here's Chrome canary

image

Chrome OS

image

abrahamjuliot avatar Aug 08 '22 06:08 abrahamjuliot

mm I wonder if brave mobile is different from normal brave in a way

I wasn't aware of jsconsole.com so I was using this for other browsers

javascript:(function () { 
    var script =  document.createElement('script');
    script.src="//cdn.jsdelivr.net/npm/eruda"; 
    document.body.appendChild(script);
    script.onload = function () { 
        eruda.init() 
    } 
})();

vis2021t avatar Aug 08 '22 06:08 vis2021t

I need to test more in Kiwi. Here's Chrome canary

Does it mean headless rtt is 0 as a special case?

I tested on Chrome, Brave, Kiwi , Chromium on both Android and Windows and Linux

All results are more than 0 in rtt normally

vis2021t avatar Aug 08 '22 07:08 vis2021t

Does it mean headless rtt is 0 as a special case?

I imagine 0 is very rare. I read somewhere that 0 was seen in some Edge browsers. Not sure if that is accurate, though. 0 could be a result of dev tools network emulation or other rare network patterns. I have a commit incoming soon that will include network info and more.

abrahamjuliot avatar Aug 08 '22 22:08 abrahamjuliot

I did some research on 192.168... and it seems to be exclusive to home WIFI networks. Something very interesting is the first set of characters following candidate:... is a hash string that actually contains the base IP address, but only on the host connection and only in Chrome and more recent versions of Safari.

Here's the ComputeFoundation function in the Chromium source code which contains this method, base_address.ipaddr().ToString().

the draft outlines the computation in greater detail (section "5.1.1.3. Computing Foundations") https://datatracker.ietf.org/doc/id/draft-ietf-ice-rfc5245bis-16.txt

abrahamjuliot avatar Aug 08 '22 23:08 abrahamjuliot

I imagine 0 is very rare. I read somewhere that 0 was seen in some Edge browsers. Not sure if that is accurate, though. 0 could be a result of dev tools network emulation or other rare network patterns. I have a commit incoming soon that will include network info and more.

Hmm what can we do I think we can take it as a suspicious point maybe

if it's unusually rare, it can be a thing, but I'm not sure if we should

it's sort of similar to :- likeHeadless one in our creepjs we can do likeUnusal or something

vis2021t avatar Aug 09 '22 06:08 vis2021t

Good idea. Added to like headless.

image

abrahamjuliot avatar Aug 09 '22 07:08 abrahamjuliot