How to customize the tokenizer default behavior?

[sincebyte]

Excellent work! It seems that NLTokenizer split text but ignored the sign char . example：emt-lib-path，the out put is emt, -lib ,-path but expect is emt ，-，lib，-，path Did there have any method to change the default behavior?

[roife]

It seems that NLTokenizer does not provide the related functionality (See https://developer.apple.com/documentation/naturallanguage/nltokenizer), so we need to handle it manually.

Actually, for English characters (or more precisely, ASCII characters), emt will call Emacs' own forward-word/backward-word.