se-scraper icon indicating copy to clipboard operation
se-scraper copied to clipboard

Published 20 hours ago • verified publisher icon NikolaiT

Safe per IP Google limits?

Open sf-steve opened this issue 5 years ago • 3 comments

Hi all, does anyone have up-to-date data for how many searches you can perform per IP per time period before getting blocked?

We have a limited scraping need, and spare server resources, so figured this would be a good solution, but would like to know how best to split the work.

I found old posts suggesting around 300 regular (not Google dorks) searches per 24 hours, but have no idea if this is still correct.

Any input greatly appreciated.

sf-steve avatar Apr 22 '19 09:04 sf-steve

I have been able to scrape for +12 hours with a 70-200 second sleep in between each query on a single IP, without detection. When I reduced the sleep range to 60-110 seconds, it was detected after ~15 minutes. Patience is key :)

YvesBos avatar Apr 23 '19 21:04 YvesBos

Thanks @YvesBos, so thats inline with the older blog posts i found. I guess we can set up a couple of VPS's and run some tests. Have you tried any tests with continuous sessions (same userdatadir) vs fresh sessions?

sf-steve avatar Apr 24 '19 13:04 sf-steve

Ah yes, I forgot to mention that. I only scrape for a couple of keywords and then relaunch the scraper with a different country in the config, thus creating a new session. I have no experience with using continuous sessions.

YvesBos avatar Apr 24 '19 13:04 YvesBos