elit icon indicating copy to clipboard operation
elit copied to clipboard

Compound words tokenization failure

Open cathxiao opened this issue 7 years ago • 1 comments

Expected Behavior

Compound words (e.g. pick-me-up, hand-me-down, know-it-all, etc.) should be tokenized as single tokens.

Actual Behavior

hyphens are treated as separators, and the components are tokenized separately.

cathxiao avatar Sep 02 '17 04:09 cathxiao