elit
elit copied to clipboard
Compound words tokenization failure
Expected Behavior
Compound words (e.g. pick-me-up, hand-me-down, know-it-all, etc.) should be tokenized as single tokens.
Actual Behavior
hyphens are treated as separators, and the components are tokenized separately.