treat icon indicating copy to clipboard operation
treat copied to clipboard

Published 20 hours ago • verified publisher icon louismullie

Tokenize Paragraph

Open peddinti opened this issue 9 years ago • 2 comments

Hi it appears i can only tokenize a sentence but not a paragraph. I have a paragraph and would like to tokenize it, how can i do that? i am also not able to get the sentences in a paragraph to be able to successively tokenize the sentences

peddinti avatar Mar 23 '15 00:03 peddinti

You would need to segment the paragraph first, and then you can tokenize the sentences.

louismullie avatar Mar 23 '15 00:03 louismullie

See the following section from the manual: https://github.com/louismullie/treat/wiki/Manual#segmenters

louismullie avatar Mar 23 '15 00:03 louismullie