smart_tokenizer_and_embedding_resize

[paulopirozelli]

Hi,

I have converted llama to HuggingFace format and now I'm trying to recover alpaca weights. However, after using the command

python weight_diff.py recover --path_raw <path_to_step_1_dir> --path_diff <path_to_step_2_dir> --path_tuned <path_to_store_recovered_weights>

it prompts the following error:

ImportError: cannot import name 'smart_tokenizer_and_embedding_resize' from 'train'

I install the libraries in requirements.txt and also installed the train library (version 0.0.5). I could not find the reason for this error. Can anyone give me a clue on that?

[rohith42]

I think you need to uninstall the train library since smart_tokenizer_and_embedding_resize is a function in train.py which is in the repo. I think pip uninstall train should do the trick.