dotnet
Special Tokens handling seems to be incorrect (at least in my scenario where I'm creating Command R+ tiktoken file from specification)
Windows 11, .NET 8 version of library 0.22.0-preview.24271.1
Describe the bug I'm trying to use tiktoken class to implement Cohere Command R+ tokenizer from the vocabulary file that cohere produces at https://storage.googleapis.com/cohere-public/tokenizers/command-r-plus.json
To Reproduce you can find a playbook here https://github.com/alkampfergit/ai-playground/blob/develop/src/python/langchainVarious/Tokenization/dotnetcohere.dib Actually I downloaded the file, then extract vocabulary node and create tiktoken file, then parse the json file to grab special tokens.
Expected behavior After I've parsed the commandR+ tokenizer specification if I tokenize a special token like <|YES_TOKEN|> it correctly recognize it as a a single token (upper part of the picture), but if I use in a sentence, like good<|YES_TOKEN|>good tokenization does not recognize the special token (lower part of the picture)
Since I've created the .tiktoken file from the specification it is possible that I've done something wrong, other tokens seems to be recognized just good.
Screenshots, Code, Sample Projects
