Google-it fails when Google returns different HTML formats

[ryudenx]

Environment

• node: v16.15.1
• npm: 8.11.0
• google-it: 1.6.3

Issue

I am suffering from a problem where using google-it fails about 50% of the time.

After investigation, I assume the following results are the root cause. Look at the following HTML pattern

success: https://github.com/ryudenx/google-it_fail/blob/master/takeda_success.html

fail: https://github.com/ryudenx/google-it_fail/blob/master/takeda_fail.html

These have different HTML format/design. Despite the same URL, same User-Agent, same IP, Google seems to return different HTML.

If possible, could you please fix this problem? Or could you support the failing HTML format?

Reproduce

Reproduced code:

const { setTimeout } = require('timers/promises');
const googleIt = require('google-it')

async function main() {
    while (true) {

        googleIt({ 'query': "takeda", 'no-display': true, 'limit': 10 }).then(results => {

            //check
            if (results.length > 1) {
                console.log("success")
            } else {
                console.log("fail")
            }

        }).catch(e => {
            console.log(e)
        })

        await setTimeout(5000);
    }
}

main()

log (example):

$ node ./test.js
success
success
success
fail
success
fail
fail

Thank you for your cooperation.

[iannortnau]

By the way it is being a gradual update for the new HTML, I start to affect me here in Brazil today, I will use your tests to try to find a solution

[iannortnau]

I updated the format of HTML, need to refine but has been working again

[iannortnau]

I made a fork with it working https://www.npmjs.com/package/google-it-reborn