Headless mode with plugin does not load target page, but "about:blank"

[dmitryp-rebel]

Below code that displays problem. When no plugin is used - everything works as expected, click on link opens and load new page. Using BAS ends with timeout error waiting while new page url is not "about:blank". It fails only in headless mode. Headless=false mode works correctly.

import puppeteer, {Browser} from "puppeteer";
import {wait} from "@rebel-internet/bot-core";
import {createPlugin, PuppeteerFingerprintPlugin} from "puppeteer-with-fingerprints";

const plugin = createPlugin({
    launch: async (options?: any): Promise<Browser> => puppeteer.launch(options),
});

(async () => {
    // Launch the browser
    const browser = await puppeteer.launch({ headless: true }); // set headless to false if you want to see the
    const browserPlugin = await plugin.launch({headless: true})
    try {
        await Promise.all([
            handle(browser, 'puppeteer').catch(e => console.error(`Failed browser`, e)),
            handle(browserPlugin, 'plugin').catch(e => console.error(`Failed plugin`, e))
        ])
    } finally {
        await browser.close();
        await browserPlugin.close();
    }
})();

async function handle(browser: Browser, prefix: string) {
    const page = await browser.newPage();

    // Set viewport size
    //await page.setViewport({ width: 1280, height: 800 });

    console.log(`${prefix}: go to`)
    // Go to the website
    await page.goto('https://www.producthunt.com/products/etsy-geeks', { waitUntil: 'networkidle2' });

    console.log(`${prefix}: click`)
    await page.click("a[href*='etsygeeks.org']")
    await wait(2000);
    try {
        await page.waitForNavigation()
    } catch (e) {}

    const pages = await browser.pages()
    const lastPage = pages.pop()
    console.log(`${prefix}: wait target url`)
    await lastPage!.waitForFunction(`!window.location.href.includes('about:blank')`, {timeout: 5_000});
    console.log(`${prefix}: Done: ` + lastPage?.url())
    console.log(`${prefix}: ` + pages.length)
}

Result:

plugin: go to
puppeteer: go to
puppeteer: click
plugin: click
puppeteer: wait target url
puppeteer: Done: https://etsygeeks.org/?ref=producthunt
puppeteer: 2
plugin: wait target url
Failed plugin TimeoutError: Waiting failed: 5000ms exceeded

Expected no errors.

[email protected]
[email protected]
[email protected]

[CheshireCaat]

Greetings, thank you, I managed to repeat the problem, unfortunately I have not yet found out the reason, as soon as I succeed, I will fix it and inform you in the ticket.

[CheshireCaat]

I found the reason - the problem is in one of the features of processing new windows created by clicks or through window.open at the custom browser engine level (only in headless mode).

This behavior is done on purpose and is handled correctly in BAS, but I missed it for the plugin.

I will try to make a fix the next time the plugins are updated with a new version of the browser, while you can use such a fix:

https://gist.github.com/CheshireCaat/4ac88c7a45e1758bccfc9b0c6b9c65fd

Sorry for the inconvenience and thank you so much for discovering this problem. I will leave your ticket open until I make a general fix at the plugin level.

[dr3adx]

I found the reason - the problem is in one of the features of processing new windows created by clicks or through window.open at the custom browser engine level (only in headless mode).

This behavior is done on purpose and is handled correctly in BAS, but I missed it for the plugin.

I will try to make a fix the next time the plugins are updated with a new version of the browser, while you can use such a fix:

https://gist.github.com/CheshireCaat/4ac88c7a45e1758bccfc9b0c6b9c65fd

Sorry for the inconvenience and thank you so much for discovering this problem. I will leave your ticket open until I make a general fix at the plugin level.

can you please write a guide explaining all settings like emulateDeviceScaleFactor and when to use such setting. The brief existing explanations are not nearly enough

Another important one: how to determine whether perfectCanvas is necessary for selected website (the actual process)

I feel like explaining these in detail will help many of us, thank you

[CheshireCaat]

@dmitryp-rebel good day.

The issue has been fixed on the engine side, now we need to wait for the engine update, then I'll update the libraries.

[CheshireCaat]

Fixed in the latest version.