media-crawler
media-crawler copied to clipboard
When parsing Washington Post articles, handle old format as well as new format
While testing the crawler with the Washington Post parser, I noticed that some errors thrown were for not being able to find the article body in articles from the Washington Post domain. This means that a different article format was present and the parser couldn't grab the article body with the given xpath
. (See Scrapy docs here for info on xpath selectors)
This article about Herman Cain, for instance, is still present in an older, wordpress-based format.
The way to go about handling this will be to extend the existing parser function to try various xpath
or css
selectors (see doc link above) to determine what format the article is in. In this instance, the article itself is now in a <div ...>
item rather than an <article itemprop="articleBody">
:
<div class="wp-column ten margin-right main-content">