web_scraper icon indicating copy to clipboard operation
web_scraper copied to clipboard

The output is different to what I see in the browser

Open theRealBitcoinClub opened this issue 4 years ago • 1 comments

When I visit a page in the browser with that following URL then I get a different output as to when I use the Scraper getPageContent() method

e.g. view-source:https://www.google.com/search?q=

delivers a different output than

webScraper.getPageContent().toString()

Is that because of some USER-AGENT settings and Google is preventing the scrape? How can I solve that issue?

theRealBitcoinClub avatar Dec 24 '21 15:12 theRealBitcoinClub

Yes probably, I have to add an option to set custom USER-AGENT.

tusharojha avatar Dec 31 '21 06:12 tusharojha