boilerpipe
boilerpipe copied to clipboard
Published
20 hours ago •
k-bx
k-bx
Library does not produce same results as http://boilerpipe-web.appspot.com/
1) Go to http://boilerpipe-web.appspot.com/
2) Type in http://arstechnica.com/ as the URL.
3) Use article extractor and HTML (extract fragment)
4) See a nice list of articles on that page
Compare to:
1) Download latest boilerpipe svn.
2) Use the following code:
final URL url = new URL("http://arstechnica.com/");
final ArticleExtractor articleExtractor = ArticleExtractor.INSTANCE;
final HTMLHighlighter htmlHighlighter = HTMLHighlighter.newExtractingInstance();
final String xhtml = htmlHighlighter.process(url, articleExtractor);
3) xhtml only contains 1 article.
Are there settings that need to be changed? Or is there a code update that
hasn't been checked in?
Original issue reported on code.google.com by [email protected] on 30 Mar 2012 at 2:50
It looks like the issue is the KeepLargestBlockFilter which rejects every block
except the largest. While taking out this filter in the library should return
results closer to http://boilerpipe-web.appspot.com/, it looks like that will
cause some non-article things that are rejected by
http://boilerpipe-web.appspot.com/ to make it through.
Is there anyplace I can go to see what filters are being by the article
extractor used by http://boilerpipe-web.appspot.com/?
Original comment by [email protected] on 30 Mar 2012 at 4:46
I'm also unable to get the same results using the HTMLHighlighter in extraction
mode. The web API (http://boilerpipe-web.appspot.com) clearly states that:
"This Web Application probably uses a more recent version than the released
versions in the Boilerpipe Google Code page. You might thus get slightly
different (hopefully better) results."
In the web demo, are you using the HTMLHighligher to extract HTML, or have you
added a different approach to boilerpipe-core? If changes have been made to
core, I'd be happy to contribute to the open source version. Hints would be
appreciated =)
Original comment by [email protected] on 19 Oct 2012 at 5:49
I got the similar issue.
When trying the URL "http://www.hokkaido-np.co.jp/news/donai/424760.html"
With ArticleExtractor and "Plain Text" output
Library code did not produce same results as http://boilerpipe-web.appspot.com/
Original comment by [email protected] on 6 Dec 2012 at 12:02
Building from source will fix this issue.
Original comment by [email protected] on 24 Jan 2013 at 8:33
I have done a build from SVN. Still Japanese URL gives bad results. Same works
gr8 on the Web App. Code snippet below:
URL url = new URL("http://d.hatena.ne.jp/mkusunok/20130817/p1");
String text = ArticleExtractor.INSTANCE.getText(url);
Original comment by [email protected] on 7 Oct 2013 at 7:24