Workbench icon indicating copy to clipboard operation
Workbench copied to clipboard

Resolve tokenization issues causing BitFunnel parser crashes

Open hausdorff opened this issue 9 years ago • 0 comments

The corpus as processed by the current version of Workbench contains characters (mostly punctuation) that cause the BitFunnel parser to crash. This commit will cause Workbench to handle these cases correctly.

There are 2 issues at the root of this problem: first, the Lucene analyzer (which we use to generate the BitFunnel chunk files) attempts to preserve URLs, and so colons are not removed from the middle of a term such as Wikipedia:dump. This causes our parser to crash. Since Luecene does remove the colon when it does not seem to appear in a URI, we simply have removed colons from all terms.

Second, we are not using the Lucene tokenizer to process article titles. This leaves a wide variety of puntuation in the corpus which crashes the tokenizer. In the new version of the corpus, the title is tokenized to avoid such problems.

hausdorff avatar Oct 28 '16 20:10 hausdorff