Support TikToken

Open abhishekbhakat opened this issue 5 months ago • 1 comments

Kimi K2 Instruct uses a tiktoken tokenizer. There are many models that supports that. It would be a nice to have model.

Jul 14 '25 17:07 abhishekbhakat

Current models are practically context-infinite, so the importance of tokenizers are not as high.

Back in a day we had to pack everything to 16K tokens so it was important. Now you can divide chars/3 and here is your token estimation 😁

Jul 15 '25 10:07 olegklimov