langutils icon indicating copy to clipboard operation
langutils copied to clipboard

Period not correctly tokenized?

Open huangjs opened this issue 12 years ago • 2 comments

Here's an example:

LANGUTILS> (tokens-for-ids (vector-document-words (vector-tag "Hello world. I'm here."))) ("Hello" "world." "I" "'" "m" "here.")

I think it should be:

("Hello" "world" "." "I" "'" "m" "here" ".")

huangjs avatar Aug 12 '11 06:08 huangjs