Tokenizer icon indicating copy to clipboard operation
Tokenizer copied to clipboard
verified publisher icon microsoft

Support Llama3.1 tokenization

Open joyceerhl opened this issue 1 year ago • 0 comments

Llama3.1's tokenizer is also BPE-based. We already ship this library in VS Code's Copilot Chat extension and it would be great if Llama3.1 was supported so that we do not need to ship another library.

joyceerhl avatar Nov 12 '24 21:11 joyceerhl