tokenizer
tokenizer copied to clipboard
Initialization options are ignored and additional pre and post splitters could be utilized
The initialization options are ignored.
The following splitters could be incorporated:
‘, ’, ”, `