se-scraper
se-scraper copied to clipboard
Set different geolocation for every request
I need to set different gelocation for every search query to Google. I set manually permissions in Browser.js (puppeteer-cluster catalog) because in se-scraper I don’t have access to browser object (or maybe I’m wrong?)
context = yield chrome.createIncognitoBrowserContext();
context.clearPermissionOverrides();
context.overridePermissions('https://www.google.pl', ['geolocation']);
Next I overwritten GoogleScraper due to extends search_keyword method with geolocation params:
async search_keyword(keyword) {
await this.page.setGeolocation({
latitude: latVar,
longitude: longVar
});
const input = await this.page.$('input[name="q"]');
await this.set_input_value(`input[name="q"]`, keyword);
await this.sleep(50);
await input.focus();
await this.page.keyboard.press("Enter");
}
I noticed that it’s necessary to reload the page to update Google results with new geo. When I put this code after Enter
in search_keyword
await this.page.evaluate(() => {
location.reload(true)
});
I got:
Error: Execution context was destroyed, most likely because of a navigation
I have also tried with put
await input.focus();
await this.page.keyboard.press("Enter");
but the same as above.
How to properly refresh the page or how to get the results related with geo params?
Same issue as above.
Honestly I don't know what the solution is for this. I don't have currently the time to debug this, can anyone step in?