puppeteer-extra
puppeteer-extra copied to clipboard
berstend
[Bug] Sites with hard challenges of CloudFlare do not work with `browser.newPage()` and call `browser.pages()` Disrupted them tabs!
The site https://www.000webhost.com/cpanel-login has a hard challenge for Cloudflare and does not open normal. So I used the puppeteer-extra-plugin-stealth plugin.
This site will not be opened with the browser.newPage(). (The tick we hit again the same challenge page)
But in the first default tab, which is always open, this site opens! (We tick and the site opens.)
This is strange so far, but it gets strange.
So I tried to use the same tab without newPage().
I tried to get the pages first:
var pages = await browser.pages();
Then open the site in the first tab with pages[0].goto.
But this time the site didn't open! (I mean fails to dissolve challenge Cloudflare)
It looks like it doesn't open when I call newPage().
And also when the pages() method is called, all tabs for this site are disrupted. (Even the first tab that can normally open this site)
I was confused and I think there's a bug here.
Sample code:
import puppeteerExtra from 'puppeteer-extra';
import StealthPlugin from 'puppeteer-extra-plugin-stealth';
var puppeteer = puppeteerExtra.use(StealthPlugin());
var browser = await puppeteer.launch({ headless: false });
// const page = await browser.newPage();
var pages = await browser.pages();
pages[0].goto('https://www.000webhost.com/cpanel-login');
Versions:
node v19.7.0
[email protected]
[email protected]
[email protected]
Chrome Version 116.0.5845.141
Video:
https://github.com/berstend/puppeteer-extra/assets/246721/f527b517-5fb5-4941-8c81-1bd37d9b1046
You can give it a try. Await puppeteer.launch ({userDataDir: path.join (os.homedir (), '.aaa-data'),} add a unified cache folder, then all tab pages that jump to the same address can share a cache.
@mowatermelon Thanks for your answer.
It wasn't bad as a temporary trick. But it has problems and of course the bug still exists.
By setting userDataDir, we can maintain the previous status and after solving the challenge manually, the next time Chrome opens, we can open the site without the challenge page.
But when the CloudFlare cookie session expires, everything goes back to the way it was before. If we use, for example, newPage(), the challenge will not be solved. Or if we reach the first tab with the help of pages()[0], everything is broken there and the site's challenge is not solved in any way.
Unless we repeat the trick again, i.e. temporarily remove the call to pages() and run Chrome once to pass the challenge. Then return that function to our code.
In general, it did not cure much pain!
I'm experiencing the same thing on other sites. What seems to be happening from my analysis, is the initial tab is "untouched" by puppeteer. So I can do whatever I want in the initial tab and I'm not detected. If I have puppeteer open it's own tab, and I manually take all actions in that tab, then I get detected as a bot. Alternatively, if I have puppeteer make the initial tab active then I take manual action, I'm detected as a bot.
So it seems any tab that gets "touched" by Puppeteer, gets detected somehow. When I look at Browser Fingerprinting, this seems to be the case. I'm not sure what they're using to detect Puppeteer even with stealth enabled.
creepjs (https://abrahamjuliot.github.io/creepjs/) is detecting puppeteer when you use any tab besides the start-up tab.
Take note that 205 lies are being detected, here's a sample:
I'm launching with the latest version of Google Chrome(Version 117.0.5938.92 (Official Build) (64-bit)) on Windows10 and am using the latest Puppeteer Node.js version alongside the stealth plugin with all evasions active.
Another creepjs image:
Any ideas about how Cloudflare is detecting Puppeteer?
I don't know the ins and outs of Puppeteer but I know they use the chrome dev tools protocol, the same as chrome dev tools.
However if I open https://nowsecure.nl/ with chrome dev tools open I can get through fine?
Unfortunately, this problem was very serious and acute.
And I had to use the service FlareSolverr. This is a proxy to bypass cloudflare and they use selenium.
I just send my first page to it and return only the cf_clearance cookie, set it to my puppeteer and continue...
Puppeteer can be detected by https://abrahamjuliot.github.io/creepjs/ if that helps. It's not just that it detects a bot, it detects PUPPETEER.
Unfortunately, this problem was very serious and acute. And I had to use the service FlareSolverr. This is a proxy to bypass cloudflare and they use selenium. I just send my first page to it and return only the
cf_clearancecookie, set it to my puppeteer and continue...
For now this is a solution with no alternatives. Puppeteer or playwright + any anti-detection methods cannot solve problems with cloudflare.
I'll make a clarification. If your ip or proxy is not on the list of suspicious ones, then there is an option to get challenge v1, which can be completed without a click and with the launch command await puppeteer.launch({ targetFilter: (target) => !!target.url() }); But if your ip is suspicious, then targetFilter: (target) => !!target.url() blocks work with the iframe and the possibility of manipulating the challenge checkbox disappears.
How is it that FlareSolverr works? Surely if Puppeteer is detected by Cloudflare then so would Selenium
How is it that FlareSolverr works? Surely if Puppeteer is detected by Cloudflare then so would Selenium
FlareSolverr uses undetected-chromedriver. UC uses completely different detection bypass methods, which are still difficult to replicate in puppeteer and puppeteer-extra.
Hello, https://www.npmjs.com/package/cloudflare-scraper With this bookshelf you can scrape and get cookies.
Hello, https://www.npmjs.com/package/cloudflare-scraper With this bookshelf you can scrape and get cookies.
Unfortunately, they have not yet implemented proxies. But this project can be copied and a proxy can be added.
Any updates on this? Does anyone understand how creepjs detects Puppeteer and what could be done to avoid detection from the Puppeteer source itself?
@Hillcow @joeledwardson @ergcode @NabiKAZ @wlc108 @mowatermelon
https://www.npmjs.com/package/puppeteer-real-browser
I had this problem too, so I had to find a solution and publish it. I found a way to make the browser look real. I connected to the browser with Puppeteer. I exported the browser and page created with Puppeteer. You can use all the functions you use with Puppeteer here. Proxy supported.
This doesn't make sense to me, I am using puppeteer to connect to chrome on android just connecting through the dev tools port, and even pages not created by puppeteer, as soon as they are "touched" cannot get past cloudflare
This doesn't make sense to me, I am using puppeteer to connect to chrome on android just connecting through the dev tools port, and even pages not created by puppeteer, as soon as they are "touched" cannot get past cloudflare
I have tried the constant page refresh problem when trying to switch to Cloudflare on the aforementioned sites. I have been repeatedly signing up on a different site for about 10 hours and they use cloudflare premium captcha. It passed all of them without any problem without needing to touch it. I don't see any problem in the package right now except fingerprint. Could you please try the package? I tried it on many sites like 000webhost openai etc. It was successful on all of them.
Unfortunately, this won't work for my use case as I launch chrome on android via ADB, forward the chrome dev tools port and then connect, so no launching is involved.
Hence why I am not sure this has fixed the issue, the only change I can see from the code in connecting is setting the user agent. If it works this is good news though
Unfortunately, this won't work for my use case as I launch chrome on android via ADB, forward the chrome dev tools port and then connect, so no launching is involved.
Hence why I am not sure this has fixed the issue, the only change I can see from the code in connecting is setting the user agent. If it works this is good news though
The important part to make it work is the cdp session which starts chrome on a port. We start a real browser on a port on the computer and connect with puppeteer's connect feature. The user agent is set to be the same as the created browser's agent. Here are some examples of it working: https://www.youtube.com/watch?v=OgQOaVNTPa4 https://www.youtube.com/watch?v=vfzEHsoJpuw https://www.youtube.com/watch?v=iTSVrtf8xXI
I will add linux support soon. If there is a site you want me to try, I can try it. In your case, after switching to cloudflare, the cookie can be taken and used with adb.
you guys can back to puppeteer version 5.5.0, all the problems will be solved, however, it quite old
you guys can back to puppeteer version 5.5.0, all the problems will be solved, however, it quite old
Can I have your example package.json?
"puppeteer": "5.5.0", you should change to this version, I will share the detail problems later.
"puppeteer": "5.5.0", you should change to this version, I will share the detail problems later.
Why that version specifically?
@nhhoang Can you provide more information and tutorial on how you made it work? Thank you!
"puppeteer": "5.5.0", you should change to this version, I will share the detail problems later.
Why that version specifically?
I can confirm that it works also on version "puppeteer": "^9.1.1". Anything higher than that does not work for me
"puppeteer": "5.5.0", you should change to this version, I will share the detail problems later.
Why that version specifically?
I can confirm that it works also on version "puppeteer": "^9.1.1". Anything higher than that does not work for me
Does not work for me. Did you downgrade anything else too, like the stealth package or the puppeteer-core package?
@smillwith61 After downgrading to "puppeteer": "^9.1.1", Do not use puppeteer-stealth package. Use the first tab to do your stuff.
let pages = await browser.pages();
let page = pages[0]
has anyone found solutions to this? I can't bypass captcha on cloudflare anymore
joeledwardson
Unfortunately, this won't work for my use case as I launch chrome on android via ADB, forward the chrome dev tools port and then connect, so no launching is involved. Hence why I am not sure this has fixed the issue, the only change I can see from the code in connecting is setting the user agent. If it works this is good news though
The important part to make it work is the cdp session which starts chrome on a port. We start a real browser on a port on the computer and connect with puppeteer's connect feature. The user agent is set to be the same as the created browser's agent. Here are some examples of it working: https://www.youtube.com/watch?v=OgQOaVNTPa4 https://www.youtube.com/watch?v=vfzEHsoJpuw https://www.youtube.com/watch?v=iTSVrtf8xXI
I will add linux support soon. If there is a site you want me to try, I can try it. In your case, after switching to cloudflare, the cookie can be taken and used with adb.
My case is that I am detected as bot when I run the command via puppeteer "await browser.pages() ;", i.e. whenever puppeteer starts touching the page. I have already used puppeteer-extra-plugin-stealth but still nothing changed.
It seems that your puppeteer-real-browser is not solving issue like mine?
