Extra entries with full text in plain_text list

[malicialab]

Some HTML files produce extra entries in the plain_text key of the JSON with the full key, in addition to the entries with the text of each paragraph, i.e., the same paragraph will appear as an entry and as part of this extra entries. This behavior only manifests using Readibility.js. Using the Python based parser this does not happen.

I am attaching one HTML file that shows this behavior:

readabilipy -V 0.2.0

readabilipy -i ef94fca40c96ebf85c2217855fe6382364b75da0d8029be5ee395f607886bd9e.html -o tmp.json

The first entry in tmp.json plain_text field has the full text, other entries have the subset per-paragraph text

readabilipy -i ef94fca40c96ebf85c2217855fe6382364b75da0d8029be5ee395f607886bd9e.html -o tmp2.json -p

tmp2.json does not have the extra entry in the plain_text field

I am wondering if this would disappear by using the latest Readibility.js instead of the embedded version. Any chance that pull request #95 is going to be incorporated soon? It would be great to avoid reporting issues already fixed in the latest Readability.js

Thanks!

ef94fca40c96ebf85c2217855fe6382364b75da0d8029be5ee395f607886bd9e.html.gz

[tpai]

I found an interesting situation. There are two different outputs between MacOS and Linux:

• When I used MacOS with Readabilipy 0.2.0 and Node.js 18, it worked without extra entries.
• When I used Docker to wrap the Python application with Linux, Readabilipy 0.2.0 and Node.js 18, the extra entries poped out.

And this library seems no longer to be maintained, so I workaround this bug by using dictionary.

article = readabilipy.simple_json_from_html_string(req.text, use_readability=True)
text_array = [obj['text'] for obj in article['plain_text']]
article_content = list(dict.fromkeys(text_array))

Hope this can help people who see this comment.