Yakhyokhuja Valikhujaev comments

Repositories
Issues
Comments

Results 22 comments of


                                            Yakhyokhuja Valikhujaev

How to build a custom tokenizer on top of a exsiting Llama 3.2 tokenizer?

I tried to do it using BPETrainer but then found that in SentencePieceBPETokenizer does not accept trainer object. Currently, this is how my `tokenizer.json` and llama 3 tokenizer json look...

new reproduced version of this repo with several new backbones and weights of them

hi @biubug6 , thanks for your great work. I recently noticed that the confidence scores are really high which may lead some unfaroble results when we need to have some...