tner icon indicating copy to clipboard operation
tner copied to clipboard

Prevent punctuation to be included in tokens

Open mirix opened this issue 1 year ago • 0 comments

Hello,

I have notice that if a work ends with a punctuation mark, the punctuation mark is included as as part of the token.

Is there a way to prevent this behaviour?

Best,

Ed

mirix avatar Nov 09 '23 13:11 mirix