HTTP Error 429: Too Many Request. Limit on total number of requests?

[bixing-udemy]

I keep meeting such error after about 100 requests.

Error accessing: http://www.google.com/search?nl=en&q=Wikipedia+Music+Business+Management&start=0&num=10&tbs= HTTP Error 429: Too Many Request

Is there a limit on the total number of requests for google search? Does the fake_useragent malfunction?

Previously, around August, I can make like 1200 requests, as long as there is a time.sleep(2) between different requests. But now, it doesn't work well anymore.

[kulla]

Actually I just ran 120 calls with time.sleep(2) and I had no problems.

[kulla]

Okay, I now had the same error. However I found the following comment in the source code at https://github.com/kulla/Google-Search-API/blob/5dbede3bdef5a1dc822c27e0e4dde94235035441/googleapi/modules/utils.py#L81-L88

# @author JuaniFilardo:
# Workaround to switch between http and https, since this maneuver
# seems to avoid the 503 error when performing a lot of queries.
# Weird, but it works.
# You may also wanna wait some time between queries, say, randint(50,65)
# between each query, and randint(180,240) every 100 queries, which is
# what I found useful.

I hope it helps :smile:

[hubshashwat]

I feel there is also a limit on the size of the query in the request. @kulla This comment of yours saved hours. Thanks!

[batman616]

@kulla Can you tell where i have to add randint(50,65) to solve this problem ? im just beginer, thanks you.

[batman616]

im trying like this, it is correct ?

    time.sleep(randint(50,65))
    https = int(time.time()) % 2 == 0
    bare_url = u"https://www.google.com/search?" if https else u"http://www.google.com/search?"
    url = bare_url + params

[kulla]

im trying like this, it is correct ?

Yes, time.sleep(randint(50,65)) seems to be fine inbetween the calls to Google. You may need https://github.com/abenassi/Google-Search-API/pull/93 as well since it is not yet merged. However Google might have changed its design once again so the code need to be changed as well...