RedditScore icon indicating copy to clipboard operation
RedditScore copied to clipboard

[Bug] Incorrect results with some numbers

Open OlehOnyshchak opened this issue 5 years ago • 0 comments

Steps

from redditscore.tokenizer import CrazyTokenizer

tokenizer = CrazyTokenizer(hashtags='split')
tokenizer.tokenize("#20yearsago")

Actual Result

['2', '0', 'y', 'e', 'a', 'r', 's', 'a', 'g', 'o']

Expected Result

['20', 'years', 'ago']

OlehOnyshchak avatar Dec 19 '19 19:12 OlehOnyshchak