crawler-google-places icon indicating copy to clipboard operation
crawler-google-places copied to clipboard

Improve the coverage rate by correct viewport & map polygon splitting

Open metalwarrior665 opened this issue 2 years ago • 0 comments

This will be more of a long play & try optimization. The result should be more places & shorted/cheaper runtime, especially with search terms and high zoom. The idea is this:

  1. Currently, the viewport is limited to 800x800 px. With the right search bar, this is more like 400x800 px. There is actually no need to keep a low viewport (it was done for predictable polygon splitting but doesn't make that much sense). To make the scraper more optimal, we need to increase the viewport as much as we can so each page captures as much area.

Then the logic should differ depending if we use search terms or scrape all places on the map

  1. For scraping with search terms, we can make the polygon splitting significantly sparser. Google gives you up to 120 places per search and will give you places outside of the visible screen. So the splits don't have to align with each other. The optimal ratio will need to be tested with various run configurations.
  2. If we scrape places without search term, we need the screens to align perfectly because we only extract the visible pins on the map. We also don't get to scroll to 120 places per page so higher zoom and larger screen make even more sense here.

metalwarrior665 avatar Sep 21 '22 17:09 metalwarrior665