SerpScraper icon indicating copy to clipboard operation
SerpScraper copied to clipboard

New Google Cookie popup is Stopping this Working

Open paulfeakins opened this issue 3 years ago • 5 comments

It seems like a new Google Cookie popup is stopping this working, see screenshot. image

That's what you get when you echo the response.

paulfeakins avatar Apr 12 '21 15:04 paulfeakins

Noticed exactly the same thing, and have just been doing some testing, and got around the problem by setting a cookie that states the cookie policy has been agreed - i.e.

$google->setPreference('results_per_page', 100);
$google->setPreference('google_domain', 'google.co.uk');

// Set A Cookie To State GDPR acceptance
$browser = $google->getBrowser();
$browser->setCookies( ".google.co.uk\tTRUE\t\\\tTRUE\t2145916800\tCONSENT\tYES+shp.gws-20210421-0-RC1.en+FX+836\r\n" );

Then do your standard do while resonse

JamesSimpson avatar Apr 29 '21 23:04 JamesSimpson

@JamesSimpson

Although the rendered page has a Cookie Popup modal, the SERP results still exist in the DOM *

The inability to parse the results is a failing of the preg_match regex formula this library uses, setting cookie fixes the symptoms but not the cause.

  • although this does depend on your user agent string, I've tested a great number of them and some return a dom with results + pop up and some return just the pop up.

LunarDevelopment avatar May 06 '21 13:05 LunarDevelopment

So for now this still seems broken and unusable, even if you add the cookies manually?

paulfeakins avatar May 27 '21 15:05 paulfeakins

So for now this still seems broken and unusable, even if you add the cookies manually?

I can't replicate any of this from my end.

That's what you get when you echo the response.

that may be, but the $searchResponse->results should still return an array of all the results for you. Is that array empty on your end? Post your code and the outputs you're getting

Athlon1600 avatar May 27 '21 21:05 Athlon1600

that may be, but the $searchResponse->results should still return an array of all the results for you. Is that array empty on your end? Post your code and the outputs you're getting

This only happens when you using EUROPE Based proxy or IP.!

sanishan avatar Jun 03 '21 12:06 sanishan