Feature Request: Customizable Word Tokenizers - Spacy

[sai-prasanna]

Spacy has customizable word level tokenizers with rules for multiple languages. I think porting that to rust would add nicely to this package. Having a customizable uniform word level tokenization across platforms (client web, server) and languages would be beneficial. Currently, idk any clean way or whether it's even possible to write bindings for spacy cython.

Spacy Tokenizer Code

https://github.com/explosion/spaCy/blob/master/spacy/tokenizer.pyx

Tokenizer exceptions for english

https://github.com/explosion/spaCy/blob/master/spacy/lang/en/tokenizer_exceptions.py

I can put in some time doing this.

[jxmorris12]

are you working on this @sai-prasanna?

[sai-prasanna]

@jxmorris12 Haven't started it yet, want to collabrate? I cannot commit much time this month on coding. But can help out reviewing, brainstorming etc.