simple-llm-finetuner
simple-llm-finetuner copied to clipboard
lxe
`LLaMATokenizer` vs `LlamaTokenizer` class names
Running inference gives the following warning:
Loading tokenizer...
The tokenizer class you load from this checkpoint is not the same type as the class this function is called from. It may result in unexpected tokenization.
The tokenizer class you load from this checkpoint is 'LLaMATokenizer'.
The class this function is called from is 'LlamaTokenizer'.
Is it a problem?
No this happens because the decapoda-research version of the llama model on huggingface used a pre-merge version of transformers where the capitalization was different. That's why you can't use AutoModel with llama
I was wasn't getting a crash due to this, it was just a warning.
i tried to use another llama7b_hf model but the same happens.... unfortunately its impossible to find out why or how it crashed.... verbose is not possible, is it?
Solution is to change the word LLaMATokenizer to LlamaTokenizer in the directory ./cache/huggingface/hub and all the subfiles (only in two if I remember well)
