Google standard search takes over 3 seconds to return answer

[SearchPirat]

Hey abenassi,

I tried your code and I'm very happy with it. I only have the issue, that each google request takes at minimum 3s. I'm using the function as follows:

import urllib.request as urllib2 from google import google

num_pages =1 Google_result = google.search(query, num_pages, lang = 'de', area = 'de')

I would be very happy if you could help me.

Best regards, SearchPirat

[JuaniFilardo]

Hi @SearchPirat!

As far as I'm concerned, that's the time that each requests takes; I'm not pretty sure what could be done to reduce it (maybe some refactoring of the API itself?). I mean, scraping the whole html returned by Google and processing it takes time, so it's not that you're doing something wrong.

Have a nice day!

PS: the area param is no longer working, check issue 52 for further information.

[SearchPirat]

Hi @JuaniFilardo Thank you for your fast reply. Good to know, that there is nothing that I'm doing wrong. I just thought it may be possible to limit the search results or the time for searching. This would also limit the time it would take to scrape the html results.

Have a nice one SearchPirat

[JuaniFilardo]

By default, the search method will return the ten results from the first page from Google. Would you want to get, for example, just the first 5 results?

[SearchPirat]

If this would speed the search up by a factor of 2 it would be a possible solution.

[SearchPirat]

@JuaniFilardo Is this possible?

[JuaniFilardo]

Um, I really don't know if the improvement would be that significant. Maybe you could try hardcoding the library in order to fetch fewer results, and then measuring the mean time it takes.

[lxwang42]

ran a quick test, lxml reduces the overall time to less than 0.8 sec(location: DE, VPS: Vultr)