GoogleScraper icon indicating copy to clipboard operation
GoogleScraper copied to clipboard

Can't Scrape News Type

Open xanderdunn opened this issue 9 years ago • 4 comments

Running this command:

GoogleScraper -m http --keyword "apple" --search-type "news" --output-filename output.json -v2

produces this error:

Traceback (most recent call last):
  File "/usr/local/bin/GoogleScraper", line 9, in <module>
    load_entry_point('GoogleScraper==0.1.37', 'console_scripts', 'GoogleScraper')()
  File "/usr/local/lib/python3.4/site-packages/GoogleScraper/core.py", line 358, in main
    scrape_jobs = parse_all_cached_files(scrape_jobs, session, scraper_search)
  File "/usr/local/lib/python3.4/site-packages/GoogleScraper/caching.py", line 413, in parse_all_cached_files
    serp = parse_again(fname, job['search_engine'], job['scrape_method'], job['query'])
  File "/usr/local/lib/python3.4/site-packages/GoogleScraper/caching.py", line 443, in parse_again
    query=query
  File "/usr/local/lib/python3.4/site-packages/GoogleScraper/parsing.py", line 1003, in parse_serp
    parser = parser(query=query)
  File "/usr/local/lib/python3.4/site-packages/GoogleScraper/parsing.py", line 419, in __init__
    super().__init__(*args, **kwargs)
  File "/usr/local/lib/python3.4/site-packages/GoogleScraper/parsing.py", line 89, in __init__
    self.__class__.__name__
AssertionError: search type "news" is not supported in GoogleParser

However, it should be supported, right? It's listed in the help message for the supported types:

GoogleScraper - WARNING - Invalid search type! Select one of ('normal', 'video', 'news', 'image')

It looks like it's not listed as a supported search type for the Google Parser

Is news scraping not yet supported, despite being listed?

xanderdunn avatar Jul 04 '15 08:07 xanderdunn

agree. not able to scrap google news yet.

steve21124 avatar Jul 10 '15 17:07 steve21124

Kindly fix this

Vineeth-Mohan avatar Apr 12 '16 05:04 Vineeth-Mohan

I am interested in implementing this.

JimmySpivey avatar Apr 04 '17 17:04 JimmySpivey

Any updates on this so far?

metromark avatar Dec 21 '18 12:12 metromark