RedditScore
RedditScore copied to clipboard
[Bug] Incorrect results with some numbers
Steps
from redditscore.tokenizer import CrazyTokenizer
tokenizer = CrazyTokenizer(hashtags='split')
tokenizer.tokenize("#20yearsago")
Actual Result
['2', '0', 'y', 'e', 'a', 'r', 's', 'a', 'g', 'o']
Expected Result
['20', 'years', 'ago']