pattern
pattern copied to clipboard
Unexpected tokenization
Hello,
This is a peculiar case of enumeration where the tokenization gives inappropriate results.
en.tokenize('See Section 3.) Or Section 2.)') results in ['See Section 3 .', ') Or Section 2 .', ')'] while the expectation is ['See Section 3 . )', 'Or Section 2 . )']