rita icon indicating copy to clipboard operation
rita copied to clipboard

Published 20 hours ago • verified publisher icon dhowe

Problems with tokenizing hyphenated words

Open dhowe opened this issue 3 years ago • 20 comments

oft-cited off-site deeply-nested

~should be handled as 3 tokens in tokenizer~ should be handled as a single token in tokenizer

  • [x] 1. ritajs tests
  • [x] 1. ritajs fix
  • [x] 2. sync tests with java
  • [x] 3. sync fix with java

dhowe avatar Jan 29 '21 06:01 dhowe