puppeteer-extra icon indicating copy to clipboard operation
puppeteer-extra copied to clipboard

[Idea] List of detection tests in docs

Open prescience-data opened this issue 4 years ago • 7 comments

Not sure if there's any unforeseen downsides to this idea, but it would be great if there was a way for users to submit new detection tests they find so that developers can improve their Puppeteer apps by running tests against real products.

I've been working on a Plugin that runs the tests that I've identified so far but it would be good to get a more complete list of detection examples from the community.

The ones I have so far:

  • Distill Networks http://promos.rtm.com
  • Sannysoft https://bot.sannysoft.com
  • SocialNetDefender http://anonymity.space/hellobot.php
  • Are You Headless? https://arh.antoinevastel.com/bots/areyouheadless
  • Fingerprint2 https://fingerprintjs.com/demo
  • Datadome https://datadome.co
  • Recaptcha3 https://antcpt.com/eng/information/demo-form/recaptcha-3-test-score.html
  • Recaptcha https://recaptcha-demo.appspot.com/recaptcha-v3-request-scores.php
  • BrowserLeaks https://browserleaks.com/webgl
  • PixelScan https://pixelscan.net

Any others people know of would be awesome!

Submitted

  • F5 Network https://ib.bri.co.id/ib-bri (tenkuken)
  • WhiteOps https://smitop.com/post/whiteops-data (evading-bot-detection)

Edit 1:

Example of how to test against them

Here is a really basic demo of how you might test each detection:

https://github.com/prescience-data/puppeteer-botcheck

prescience-data avatar Jul 13 '20 05:07 prescience-data

F5 Network Bot Defense https://ib.bri.co.id/ib-bri/

tenkuken avatar Jul 14 '20 10:07 tenkuken

This is useful. Can someone also add the tests for WhiteOps?

https://smitop.com/post/whiteops-data/

evading-bot-detection avatar Jul 15 '20 05:07 evading-bot-detection

I like the idea but most of these sites/links are not bot detection tests? :-)

berstend avatar Jul 22 '20 08:07 berstend

Some of the products such as Datadome have no public "demos" that I know of, so the idea is to build up your tests like this:

https://github.com/prescience-data/puppeteer-botcheck

prescience-data avatar Jul 22 '20 09:07 prescience-data

PerimeterX: https://www.usa-people-search.com/names/a_1_150_0

@berstend These sites (well, most of them) are clients of bot detection companies. If you visit them with, say, window.callPhantom exposed, you will get redirected to a page that will make you do a captcha.

chris124567 avatar Aug 21 '20 14:08 chris124567

The first link one does not show Distill Networks operating anymore. But a good example is given by WesternUnion Getting a non-empty change rate in #smoExchangeRate means Distill Networks is bypassed (which currently not the case with latest stealth in headless mode)

drzraf avatar Mar 12 '21 05:03 drzraf

But a good example is given by WesternUnion

I'd say they're more on the extreme end of the gaussian curve of anti-bot 😄 image

berstend avatar Mar 12 '21 05:03 berstend

@drzraf Can you nowadays return that #smoExchangeRate?

francobasilico avatar Jan 22 '23 19:01 francobasilico