ValueError: Tokenizer class BaichuanTokenizer does not exist or is not currently imported.

[98765mm]

when i use local model, it will happen ValueError: Tokenizer class BaichuanTokenizer does not exist or is not currently imported.

so，what should i do

[tuxiangxianzhe]

I meet the same error, how should I do?

[tuxiangxianzhe]

I solved this problem. If I load tokenizer with "tokenizer = AutoTokenizer.from_pretrained("./Baichuan2-7B-Chat-4bits/", use_fast=False)", this error will happen. While if I use "tokenizer = AutoTokenizer.from_pretrained("./Baichuan2-7B-Chat-4bits/", use_fast=False, trust_remote_code=True)", it runs successfully.

[whn09]

You should use tokenizer.save_pretrained('YOUR_PATH') to save the tokenizer as well as the model.

[wangdong2023]

I solved this problem. If I load tokenizer with "tokenizer = AutoTokenizer.from_pretrained("./Baichuan2-7B-Chat-4bits/", use_fast=False)", this error will happen. While if I use "tokenizer = AutoTokenizer.from_pretrained("./Baichuan2-7B-Chat-4bits/", use_fast=False, trust_remote_code=True)", it runs successfully.

met the same problem, already has trust_remote_code as True