Can I use the transformers.AutoTokenizer to load the tokenizer?

[tian969]

I know the tokenizer.py in this Repo use TikTokenizer, can I use transformers.AutoTokenizer to load the tokenizer so that I dont need to amend my code class? And if i not use tokenizer.py, ChatFormat can not be used too.

[tian969]

I mean transformers.PretrainedTokenizer class

[ppaanngggg]

same question

[ppaanngggg]

I find the solution, you should use model files on huggingface. There is a tokenizer.json file can be loaded directly.

[subramen]

Yes, you can use AutoTokenizer.from_pretrained('meta-llama/Meta-Llama-3-8B-Instruct)