puppeteer-extra [Bug] CloudFlare protection (https://nowsecure.nl) bypass only with {headless: false}, not with {headless: true}

[Bug] CloudFlare protection (https://nowsecure.nl) bypass only with {headless: false}, not with {headless: true}

Open sngrl2 opened this issue 1 year ago • 11 comments

Describe the bug

I'm trying to bypass CloudFlare protection, using for test https://nowsecure.nl website My simple code works well when headless option is set to false, but did not work when headless is true

Command to execute

DEBUG=puppeteer-extra,puppeteer-extra-plugin:* node cloudflare_test.js

Code Snippet

const puppeteer = require('puppeteer-extra')
const StealthPlugin = require('puppeteer-extra-plugin-stealth')
puppeteer.use(StealthPlugin())
const {executablePath} = require('puppeteer')

puppeteer.launch({
    executablePath: executablePath(),
    headless: false,
}).then(async browser => {
    console.log('Running tests..')
    const page = await browser.newPage()

    await page.goto('https://nowsecure.nl')
    await page.waitForTimeout(10000)

    await page.screenshot({
        path: 'stealth_test__headless-false.png',
        fullPage: true
    })
    await browser.close()
    console.log(`All done, check the screenshot`)
})

Screenshot with result: protection is passed

stealth_test__headless-false

Debug info from console

  puppeteer-extra-plugin:base:stealth Initialized. +0ms
  puppeteer-extra plugin registered stealth +0ms
  puppeteer-extra dependencies missing Set(16) {
  'stealth/evasions/chrome.app',
  'stealth/evasions/chrome.csi',
  'stealth/evasions/chrome.loadTimes',
  'stealth/evasions/chrome.runtime',
  'stealth/evasions/defaultArgs',
  'stealth/evasions/iframe.contentWindow',
  'stealth/evasions/media.codecs',
  'stealth/evasions/navigator.hardwareConcurrency',
  'stealth/evasions/navigator.languages',
  'stealth/evasions/navigator.permissions',
  'stealth/evasions/navigator.plugins',
  'stealth/evasions/navigator.webdriver',
  'stealth/evasions/sourceurl',
  'stealth/evasions/user-agent-override',
  'stealth/evasions/webgl.vendor',
  'stealth/evasions/window.outerdimensions'
} +28ms
  puppeteer-extra-plugin:base:stealth/evasions/chrome.app Initialized. +0ms
  puppeteer-extra plugin registered stealth/evasions/chrome.app +7ms
  puppeteer-extra-plugin:base:stealth/evasions/chrome.csi Initialized. +0ms
  puppeteer-extra plugin registered stealth/evasions/chrome.csi +2ms
  puppeteer-extra-plugin:base:stealth/evasions/chrome.loadTimes Initialized. +0ms
  puppeteer-extra plugin registered stealth/evasions/chrome.loadTimes +2ms
  puppeteer-extra-plugin:base:stealth/evasions/chrome.runtime Initialized. +0ms
  puppeteer-extra plugin registered stealth/evasions/chrome.runtime +2ms
  puppeteer-extra-plugin:base:stealth/evasions/defaultArgs Initialized. +0ms
  puppeteer-extra plugin registered stealth/evasions/defaultArgs +2ms
  puppeteer-extra-plugin:base:stealth/evasions/iframe.contentWindow Initialized. +0ms
  puppeteer-extra plugin registered stealth/evasions/iframe.contentWindow +1ms
  puppeteer-extra-plugin:base:stealth/evasions/media.codecs Initialized. +0ms
  puppeteer-extra plugin registered stealth/evasions/media.codecs +2ms
  puppeteer-extra-plugin:base:stealth/evasions/navigator.hardwareConcurrency Initialized. +0ms
  puppeteer-extra plugin registered stealth/evasions/navigator.hardwareConcurrency +2ms
  puppeteer-extra-plugin:base:stealth/evasions/navigator.languages Initialized. +0ms
  puppeteer-extra plugin registered stealth/evasions/navigator.languages +2ms
  puppeteer-extra-plugin:base:stealth/evasions/navigator.permissions Initialized. +0ms
  puppeteer-extra plugin registered stealth/evasions/navigator.permissions +2ms
  puppeteer-extra-plugin:base:stealth/evasions/navigator.plugins Initialized. +0ms
  puppeteer-extra plugin registered stealth/evasions/navigator.plugins +5ms
  puppeteer-extra-plugin:base:stealth/evasions/navigator.webdriver Initialized. +0ms
  puppeteer-extra plugin registered stealth/evasions/navigator.webdriver +2ms
  puppeteer-extra-plugin:base:stealth/evasions/sourceurl Initialized. +0ms
  puppeteer-extra plugin registered stealth/evasions/sourceurl +2ms
  puppeteer-extra-plugin:base:stealth/evasions/user-agent-override Initialized. +0ms
  puppeteer-extra plugin registered stealth/evasions/user-agent-override +1ms
  puppeteer-extra dependencies missing Set(3) {
  'stealth/evasions/webgl.vendor',
  'stealth/evasions/window.outerdimensions',
  'user-preferences'
} +1ms
  puppeteer-extra-plugin:base:stealth/evasions/webgl.vendor Initialized. +0ms
  puppeteer-extra plugin registered stealth/evasions/webgl.vendor +2ms
  puppeteer-extra-plugin:base:stealth/evasions/window.outerdimensions Initialized. +0ms
  puppeteer-extra plugin registered stealth/evasions/window.outerdimensions +2ms
  puppeteer-extra-plugin:base:user-preferences Initialized. +0ms
  puppeteer-extra plugin registered user-preferences +4ms
  puppeteer-extra dependencies missing Set(1) { 'user-data-dir' } +1ms
  puppeteer-extra-plugin:base:user-data-dir Initialized. +0ms
  puppeteer-extra-plugin:user-data-dir initialized {
  deleteTemporary: true,
  deleteExisting: false,
  files: [],
  folderPath: 'C:\\Users\\Acer\\AppData\\Local\\Temp',
  folderPrefix: 'puppeteer_dev_profile-'
} +0ms
  puppeteer-extra plugin registered user-data-dir +41ms
  puppeteer-extra ignoring dependency 'stealth/evasions/webgl.vendor', which has been required already. +0ms
  puppeteer-extra ignoring dependency 'stealth/evasions/window.outerdimensions', which has been required already. +0m
s
  puppeteer-extra orderPlugins:before [
  'stealth',
  'stealth/evasions/chrome.app',
  'stealth/evasions/chrome.csi',
  'stealth/evasions/chrome.loadTimes',
  'stealth/evasions/chrome.runtime',
  'stealth/evasions/defaultArgs',
  'stealth/evasions/iframe.contentWindow',
  'stealth/evasions/media.codecs',
  'stealth/evasions/navigator.hardwareConcurrency',
  'stealth/evasions/navigator.languages',
  'stealth/evasions/navigator.permissions',
  'stealth/evasions/navigator.plugins',
  'stealth/evasions/navigator.webdriver',
  'stealth/evasions/sourceurl',
  'stealth/evasions/user-agent-override',
  'stealth/evasions/webgl.vendor',
  'stealth/evasions/window.outerdimensions',
  'user-preferences',
  'user-data-dir'
] +1ms
  puppeteer-extra orderPlugins:after [
  'stealth',
  'stealth/evasions/chrome.app',
  'stealth/evasions/chrome.csi',
  'stealth/evasions/chrome.loadTimes',
  'stealth/evasions/chrome.runtime',
  'stealth/evasions/media.codecs',
  'stealth/evasions/navigator.hardwareConcurrency',
  'stealth/evasions/navigator.languages',
  'stealth/evasions/navigator.permissions',
  'stealth/evasions/navigator.plugins',
  'stealth/evasions/navigator.webdriver',
  'stealth/evasions/sourceurl',
  'stealth/evasions/user-agent-override',
  'stealth/evasions/webgl.vendor',
  'stealth/evasions/window.outerdimensions',
  'stealth/evasions/defaultArgs',
  'stealth/evasions/iframe.contentWindow',
  'user-preferences',
  'user-data-dir'
] +1ms
  puppeteer-extra-plugin:user-preferences _userPrefsFromPlugins { intl: { accept_languages: 'en-US,en' } } +0ms
  puppeteer-extra-plugin:user-data-dir created custom dir C:\Users\Acer\AppData\Local\Temp\puppeteer_dev_profile-8DQB
U2 +15ms
  puppeteer-extra-plugin:user-data-dir Wrote file C:\Users\Acer\AppData\Local\Temp\puppeteer_dev_profile-8DQBU2\Defau
lt\Preferences +3ms
Running tests..
  puppeteer-extra-plugin:stealth/evasions/user-agent-override onPageCreated - Will set these user agent options {
  override: {
    userAgent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Saf
ari/537.36',
    platform: 'Win32',
    userAgentMetadata: {
      brands: [Array],
      fullVersion: '109.0.0.0',
      platform: 'Windows',
      platformVersion: '10.0',
      architecture: 'x86',
      model: '',
      mobile: false
    }
  },
  opts: { userAgent: null, locale: 'en-US,en', maskLinux: true }
} +0ms
  puppeteer-extra-plugin:stealth/evasions/sourceurl Stripping sourceURL { method: 'Runtime.evaluate' } +0ms
  puppeteer-extra-plugin:stealth/evasions/sourceurl Stripping sourceURL { method: 'Runtime.evaluate' } +4ms
  puppeteer-extra-plugin:stealth/evasions/sourceurl Stripping sourceURL { method: 'Runtime.evaluate' } +486ms
  puppeteer-extra-plugin:stealth/evasions/sourceurl Stripping sourceURL { method: 'Runtime.evaluate' } +2ms
  puppeteer-extra-plugin:stealth/evasions/sourceurl Stripping sourceURL { method: 'Runtime.evaluate' } +1s
  puppeteer-extra-plugin:stealth/evasions/sourceurl Stripping sourceURL { method: 'Runtime.evaluate' } +3ms
  puppeteer-extra-plugin:stealth/evasions/sourceurl Stripping sourceURL { method: 'Runtime.evaluate' } +260ms
  puppeteer-extra-plugin:stealth/evasions/sourceurl Stripping sourceURL { method: 'Runtime.evaluate' } +1ms
  puppeteer-extra-plugin:stealth/evasions/sourceurl Stripping sourceURL { method: 'Runtime.evaluate' } +518ms
  puppeteer-extra-plugin:stealth/evasions/sourceurl Stripping sourceURL { method: 'Runtime.evaluate' } +1ms
  puppeteer-extra-plugin:stealth/evasions/sourceurl Stripping sourceURL { method: 'Runtime.evaluate' } +715ms
  puppeteer-extra-plugin:stealth/evasions/sourceurl Stripping sourceURL { method: 'Runtime.evaluate' } +2ms
  puppeteer-extra-plugin:stealth/evasions/sourceurl Stripping sourceURL { method: 'Runtime.evaluate' } +440ms
  puppeteer-extra-plugin:stealth/evasions/sourceurl Stripping sourceURL { method: 'Runtime.evaluate' } +1ms
  puppeteer-extra-plugin:stealth/evasions/sourceurl Stripping sourceURL { method: 'Runtime.evaluate' } +468ms
  puppeteer-extra-plugin:stealth/evasions/sourceurl Stripping sourceURL { method: 'Runtime.evaluate' } +1ms
  puppeteer-extra-plugin:stealth/evasions/sourceurl Stripping sourceURL { method: 'Runtime.evaluate' } +471ms
  puppeteer-extra-plugin:stealth/evasions/sourceurl Stripping sourceURL { method: 'Runtime.evaluate' } +1ms
  puppeteer-extra-plugin:stealth/evasions/sourceurl Stripping sourceURL { method: 'Runtime.evaluate' } +670ms
  puppeteer-extra-plugin:stealth/evasions/sourceurl Stripping sourceURL { method: 'Runtime.evaluate' } +2ms
  puppeteer-extra-plugin:user-data-dir onDisconnected +14s
  puppeteer-extra-plugin:user-data-dir removeUserDataDir C:\Users\Acer\AppData\Local\Temp\puppeteer_dev_profile-8DQBU
2 +1ms
All done, check the screenshot
  puppeteer-extra-plugin:user-data-dir null +565ms

Then, change headless value from false to true

const puppeteer = require('puppeteer-extra')
const StealthPlugin = require('puppeteer-extra-plugin-stealth')
puppeteer.use(StealthPlugin())
const {executablePath} = require('puppeteer')

puppeteer.launch({
    executablePath: executablePath(),
    headless: true,
}).then(async browser => {
    console.log('Running tests..')
    const page = await browser.newPage()

    await page.goto('https://nowsecure.nl')
    await page.waitForTimeout(10000)

    await page.screenshot({
        path: 'stealth_test__headless-true.png',
        fullPage: true
    })
    await browser.close()
    console.log(`All done, check the screenshot`)
})

Screenshot with result: protection is NOT passed

stealth_test__headless-true

Debug info from console

  puppeteer-extra-plugin:base:stealth Initialized. +0ms
  puppeteer-extra plugin registered stealth +0ms
  puppeteer-extra dependencies missing Set(16) {
  'stealth/evasions/chrome.app',
  'stealth/evasions/chrome.csi',
  'stealth/evasions/chrome.loadTimes',
  'stealth/evasions/chrome.runtime',
  'stealth/evasions/defaultArgs',
  'stealth/evasions/iframe.contentWindow',
  'stealth/evasions/media.codecs',
  'stealth/evasions/navigator.hardwareConcurrency',
  'stealth/evasions/navigator.languages',
  'stealth/evasions/navigator.permissions',
  'stealth/evasions/navigator.plugins',
  'stealth/evasions/navigator.webdriver',
  'stealth/evasions/sourceurl',
  'stealth/evasions/user-agent-override',
  'stealth/evasions/webgl.vendor',
  'stealth/evasions/window.outerdimensions'
} +27ms
  puppeteer-extra-plugin:base:stealth/evasions/chrome.app Initialized. +0ms
  puppeteer-extra plugin registered stealth/evasions/chrome.app +6ms
  puppeteer-extra-plugin:base:stealth/evasions/chrome.csi Initialized. +0ms
  puppeteer-extra plugin registered stealth/evasions/chrome.csi +2ms
  puppeteer-extra-plugin:base:stealth/evasions/chrome.loadTimes Initialized. +0ms
  puppeteer-extra plugin registered stealth/evasions/chrome.loadTimes +2ms
  puppeteer-extra-plugin:base:stealth/evasions/chrome.runtime Initialized. +0ms
  puppeteer-extra plugin registered stealth/evasions/chrome.runtime +2ms
  puppeteer-extra-plugin:base:stealth/evasions/defaultArgs Initialized. +0ms
  puppeteer-extra plugin registered stealth/evasions/defaultArgs +2ms
  puppeteer-extra-plugin:base:stealth/evasions/iframe.contentWindow Initialized. +0ms
  puppeteer-extra plugin registered stealth/evasions/iframe.contentWindow +2ms
  puppeteer-extra-plugin:base:stealth/evasions/media.codecs Initialized. +0ms
  puppeteer-extra plugin registered stealth/evasions/media.codecs +2ms
  puppeteer-extra-plugin:base:stealth/evasions/navigator.hardwareConcurrency Initialized. +0ms
  puppeteer-extra plugin registered stealth/evasions/navigator.hardwareConcurrency +1ms
  puppeteer-extra-plugin:base:stealth/evasions/navigator.languages Initialized. +0ms
  puppeteer-extra plugin registered stealth/evasions/navigator.languages +2ms
  puppeteer-extra-plugin:base:stealth/evasions/navigator.permissions Initialized. +0ms
  puppeteer-extra plugin registered stealth/evasions/navigator.permissions +1ms
  puppeteer-extra-plugin:base:stealth/evasions/navigator.plugins Initialized. +0ms
  puppeteer-extra plugin registered stealth/evasions/navigator.plugins +6ms
  puppeteer-extra-plugin:base:stealth/evasions/navigator.webdriver Initialized. +0ms
  puppeteer-extra plugin registered stealth/evasions/navigator.webdriver +1ms
  puppeteer-extra-plugin:base:stealth/evasions/sourceurl Initialized. +0ms
  puppeteer-extra plugin registered stealth/evasions/sourceurl +2ms
  puppeteer-extra-plugin:base:stealth/evasions/user-agent-override Initialized. +0ms
  puppeteer-extra plugin registered stealth/evasions/user-agent-override +2ms
  puppeteer-extra dependencies missing Set(3) {
  'stealth/evasions/webgl.vendor',
  'stealth/evasions/window.outerdimensions',
  'user-preferences'
} +1ms
  puppeteer-extra-plugin:base:stealth/evasions/webgl.vendor Initialized. +0ms
  puppeteer-extra plugin registered stealth/evasions/webgl.vendor +1ms
  puppeteer-extra-plugin:base:stealth/evasions/window.outerdimensions Initialized. +0ms
  puppeteer-extra plugin registered stealth/evasions/window.outerdimensions +2ms
  puppeteer-extra-plugin:base:user-preferences Initialized. +0ms
  puppeteer-extra plugin registered user-preferences +3ms
  puppeteer-extra dependencies missing Set(1) { 'user-data-dir' } +0ms
  puppeteer-extra-plugin:base:user-data-dir Initialized. +0ms
  puppeteer-extra-plugin:user-data-dir initialized {
  deleteTemporary: true,
  deleteExisting: false,
  files: [],
  folderPath: 'C:\\Users\\Acer\\AppData\\Local\\Temp',
  folderPrefix: 'puppeteer_dev_profile-'
} +0ms
  puppeteer-extra plugin registered user-data-dir +42ms
  puppeteer-extra ignoring dependency 'stealth/evasions/webgl.vendor', which has been required already. +1ms
  puppeteer-extra ignoring dependency 'stealth/evasions/window.outerdimensions', which has been required already. +0m
s
  puppeteer-extra orderPlugins:before [
  'stealth',
  'stealth/evasions/chrome.app',
  'stealth/evasions/chrome.csi',
  'stealth/evasions/chrome.loadTimes',
  'stealth/evasions/chrome.runtime',
  'stealth/evasions/defaultArgs',
  'stealth/evasions/iframe.contentWindow',
  'stealth/evasions/media.codecs',
  'stealth/evasions/navigator.hardwareConcurrency',
  'stealth/evasions/navigator.languages',
  'stealth/evasions/navigator.permissions',
  'stealth/evasions/navigator.plugins',
  'stealth/evasions/navigator.webdriver',
  'stealth/evasions/sourceurl',
  'stealth/evasions/user-agent-override',
  'stealth/evasions/webgl.vendor',
  'stealth/evasions/window.outerdimensions',
  'user-preferences',
  'user-data-dir'
] +1ms
  puppeteer-extra orderPlugins:after [
  'stealth',
  'stealth/evasions/chrome.app',
  'stealth/evasions/chrome.csi',
  'stealth/evasions/chrome.loadTimes',
  'stealth/evasions/chrome.runtime',
  'stealth/evasions/media.codecs',
  'stealth/evasions/navigator.hardwareConcurrency',
  'stealth/evasions/navigator.languages',
  'stealth/evasions/navigator.permissions',
  'stealth/evasions/navigator.plugins',
  'stealth/evasions/navigator.webdriver',
  'stealth/evasions/sourceurl',
  'stealth/evasions/user-agent-override',
  'stealth/evasions/webgl.vendor',
  'stealth/evasions/window.outerdimensions',
  'stealth/evasions/defaultArgs',
  'stealth/evasions/iframe.contentWindow',
  'user-preferences',
  'user-data-dir'
] +1ms
  puppeteer-extra-plugin:user-preferences _userPrefsFromPlugins { intl: { accept_languages: 'en-US,en' } } +0ms
  puppeteer-extra-plugin:user-data-dir created custom dir C:\Users\Acer\AppData\Local\Temp\puppeteer_dev_profile-PhBm
wl +17ms
  puppeteer-extra-plugin:user-data-dir Wrote file C:\Users\Acer\AppData\Local\Temp\puppeteer_dev_profile-PhBmwl\Defau
lt\Preferences +6ms
Running tests..
  puppeteer-extra-plugin:stealth/evasions/user-agent-override onPageCreated - Will set these user agent options {
  override: {
    userAgent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.5412.0
Safari/537.36',
    platform: 'Win32',
    userAgentMetadata: {
      brands: [Array],
      fullVersion: '109.0.5412.0',
      platform: 'Windows',
      platformVersion: '10.0',
      architecture: 'x86',
      model: '',
      mobile: false
    },
    acceptLanguage: 'en-US,en'
  },
  opts: { userAgent: null, locale: 'en-US,en', maskLinux: true }
} +0ms
  puppeteer-extra-plugin:stealth/evasions/sourceurl Stripping sourceURL { method: 'Runtime.evaluate' } +0ms
  puppeteer-extra-plugin:stealth/evasions/sourceurl Stripping sourceURL { method: 'Runtime.evaluate' } +1ms
  puppeteer-extra-plugin:stealth/evasions/sourceurl Stripping sourceURL { method: 'Runtime.evaluate' } +457ms
  puppeteer-extra-plugin:stealth/evasions/sourceurl Stripping sourceURL { method: 'Runtime.evaluate' } +1ms
  puppeteer-extra-plugin:stealth/evasions/sourceurl Stripping sourceURL { method: 'Runtime.evaluate' } +2s
  puppeteer-extra-plugin:stealth/evasions/sourceurl Stripping sourceURL { method: 'Runtime.evaluate' } +2ms
  puppeteer-extra-plugin:stealth/evasions/sourceurl Stripping sourceURL { method: 'Runtime.evaluate' } +516ms
  puppeteer-extra-plugin:stealth/evasions/sourceurl Stripping sourceURL { method: 'Runtime.evaluate' } +1ms
  puppeteer-extra-plugin:stealth/evasions/sourceurl Stripping sourceURL { method: 'Runtime.evaluate' } +528ms
  puppeteer-extra-plugin:stealth/evasions/sourceurl Stripping sourceURL { method: 'Runtime.evaluate' } +1ms
  puppeteer-extra-plugin:stealth/evasions/sourceurl Stripping sourceURL { method: 'Runtime.evaluate' } +3s
  puppeteer-extra-plugin:stealth/evasions/sourceurl Stripping sourceURL { method: 'Runtime.evaluate' } +1ms
  puppeteer-extra-plugin:stealth/evasions/sourceurl Stripping sourceURL { method: 'Runtime.evaluate' } +1s
  puppeteer-extra-plugin:stealth/evasions/sourceurl Stripping sourceURL { method: 'Runtime.evaluate' } +2ms
  puppeteer-extra-plugin:stealth/evasions/sourceurl Stripping sourceURL { method: 'Runtime.evaluate' } +762ms
  puppeteer-extra-plugin:stealth/evasions/sourceurl Stripping sourceURL { method: 'Runtime.evaluate' } +4ms
  puppeteer-extra-plugin:user-data-dir onDisconnected +12s
  puppeteer-extra-plugin:user-data-dir removeUserDataDir C:\Users\Acer\AppData\Local\Temp\puppeteer_dev_profile-PhBmw
l +2ms
All done, check the screenshot
  puppeteer-extra-plugin:user-data-dir null +631ms

Only one strange thing which I see: with headless: false user-agent is Chrome/109.0.0.0, but with headless: true user-agent is Chrome/109.0.5412.0

Versions

  System:
    OS: Windows 10 10.0.19044
    CPU: (8) x64 Intel(R) Core(TM) i7-4720HQ CPU @ 2.60GHz
    Memory: 5.64 GB / 15.93 GB
  Binaries:
    Node: 14.17.0 - C:\Program Files\nodejs\node.EXE
    npm: 6.14.3 - C:\Web\project_name\node_modules\.bin\npm.CMD
  npmPackages:
    puppeteer: ^19.5.2 => 19.5.2
    puppeteer-extra: ^3.3.4 => 3.3.4
    puppeteer-extra-plugin-stealth: ^2.11.1 => 2.11.1

Jan 21 '23 20:01 sngrl2

puppeteer-extra puppeteer-extra copied to clipboard

[Bug] CloudFlare protection (https://nowsecure.nl) bypass only with {headless: false}, not with {headless: true}

puppeteer-extra
puppeteer-extra copied to clipboard