Blogger / Blogspot issue

[ontopicprojects]

Some blogspot / blogger sites don't seem to parse: here is an example:

`from newspaper import Article

url = 'http://www.righto.com/2011/07/cells-are-very-fast-and-crowded-places.html'

article = Article(url) article.download() article.parse() print(article.text)`

this prints ""

[johnbumgarner]

The primary reason that you cannot extract from this site with Newspaper is because the tags commonly queried by this module do not exist on the website http://www.righto.com. You should use the Python libraries requests and BeautifulSoup to extract the items that you want.

[AndyTheFactory]

you can achieve this with some minor changes.

1. the cleaners class must be fixed (it removes itemprop containing articleBody if it is not exactly "articleBody") see https://github.com/codelucas/newspaper/pull/953
2. extend the extractor class and replace it in your article :+1:

class myExtractor(ContentExtractor):
    def nodes_to_check(self, doc):
        generator = self.parser.getElementsByTag(doc, tag='meta', attr={'name':'generator'})
        for t in generator:
            if t.attrib.get('content') == 'blogger':
                nodes = self.parser.getElementsByTag(doc, tag='div', attr={'class':'post-body'})
                return nodes
        return super().nodes_to_check(doc)

and

a = Article('http://www.righto.com/2011/07/cells-are-very-fast-and-crowded-places.html')
a.extractor = myExtractor(a.config)
a.download()
a.parse()