media-crawler icon indicating copy to clipboard operation
media-crawler copied to clipboard

When parsing Washington Post articles, handle old format as well as new format

Open josephpd3 opened this issue 7 years ago • 0 comments

While testing the crawler with the Washington Post parser, I noticed that some errors thrown were for not being able to find the article body in articles from the Washington Post domain. This means that a different article format was present and the parser couldn't grab the article body with the given xpath. (See Scrapy docs here for info on xpath selectors)

This article about Herman Cain, for instance, is still present in an older, wordpress-based format.

The way to go about handling this will be to extend the existing parser function to try various xpath or css selectors (see doc link above) to determine what format the article is in. In this instance, the article itself is now in a <div ...> item rather than an <article itemprop="articleBody">:

<div class="wp-column ten margin-right main-content">

josephpd3 avatar Sep 30 '17 14:09 josephpd3