Open problems

[MaxDall]

Rewrite ArticleBody

• Problem: The current object implementation bundling all information about the articles' body called ArticleBody seems overly complex and error-prone, while lacking some features.
  1. Do we need a specific layout format (summary -> sub-headlines -> paragraphs) at all, since some regions like US don't use this kind of layout (see #310). This would also simplify node extraction.
  2. Support access to the 'raw' text or even lxml node on a per-element base. I.e. if we iterate over the elements of ArticleBody (alongside the nodes extracted from the HTML document) it should be possible to exact both the cleaned text and the 'raw' original text, also being able to access the lxml.html.HtmlElement would be beneficial but complicated for upcoming serialization approaches.

Write a spyder

• Problem: Fundus heavily depends on the information given by publishers to crawl articles, in this case, a map linking articles. While initially thought of as a feature to crawl publishers 'politely', adding optional functionality to process an entire domain in the form of a spyder and therefore not relying on sitemaps would be an excellent addition to Fundus.
• Requirements: The spyder should ...
  • operate on the same level as HTMLSource yielding HTML for the scrapers to scrape. This requires a rework of the naming and object hierarchy as there should be a proper interface now replacing HTMLSource which intern was used as an object and interface at the same time.
  • only operate in cases where no sitemap is provided and also not per default
  • support functionality to add delays between requests
• Expensive: There could be a central request unit organizing request delays and access rules set by the publisher (see /robots.txt) so one would send a deferred request to this unit and expect Optional[HTML] in return. This would cover all requests in all parts of Fundus.

Serialization

Should we simplify serialization at all, either through utility functions or proper serialization implementations?

Replace PublisherEnum's Enum implementation with a custom one

Python's implementation of the Enum class is a bit messy, especially typing-wise. Since Fundus' PublisherEnum is solely built around the Enum class, this led to many typing issues but some unexpected behavior as well, and thus the PublisherEnum became notorious for causing problems.