llama3 to hf model conversion does not work

[yuri-son]

trafficstars

System Info

Hello developer, The Llama-3 model was released today.

I want to convert this model to a hf model, but when I follow the readme, the following issue occurs. File "/workspace/00_develop/llama/llama-recipes/./transformers/src/transformers/models/llama/convert_llama_weights_to_hf.py", line 339, in <module> main() File "/workspace/00_develop/llama/llama-recipes/./transformers/src/transformers/models/llama/convert_llama_weights_to_hf.py", line 326, in main write_model( File "/workspace/00_develop/llama/llama-recipes/./transformers/src/transformers/models/llama/convert_llama_weights_to_hf.py", line 120, in write_model tokenizer = tokenizer_class(tokenizer_path) File "/opt/conda/lib/python3.10/site-packages/transformers/models/llama/tokenization_llama_fast.py", line 133, in __init__ super().__init__( File "/opt/conda/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 117, in __init__ slow_tokenizer = self.slow_tokenizer_class(*args, **kwargs) File "/opt/conda/lib/python3.10/site-packages/transformers/models/llama/tokenization_llama.py", line 184, in __init__ self.sp_model = self.get_spm_processor(kwargs.pop("from_slow", False)) File "/opt/conda/lib/python3.10/site-packages/transformers/models/llama/tokenization_llama.py", line 217, in get_spm_processor model = model_pb2.ModelProto.FromString(sp_model) google.protobuf.message.DecodeError: Error parsing message

I would really appreciate it if you could give me some guidance on how to solve this problem. Please help me. thank you!!!

Information

• [X] The official example scripts
• [ ] My own modified scripts

🐛 Describe the bug

'python ./transformers/src/transformers/models/llama/convert_llama_weights_to_hf.py
--input_dir /workspace/00_develop/llama/llama3/llama3-model/Meta-Llama-3-8B
--model_size 7B
--output_dir /workspace/00_develop/llama/llama3/hf-llama3-model/hf_Meta_Llama_3_8B'

Error logs

raceback (most recent call last): File "/workspace/00_develop/llama/llama-recipes/./transformers/src/transformers/models/llama/convert_llama_weights_to_hf.py", line 339, in main() File "/workspace/00_develop/llama/llama-recipes/./transformers/src/transformers/models/llama/convert_llama_weights_to_hf.py", line 326, in main write_model( File "/workspace/00_develop/llama/llama-recipes/./transformers/src/transformers/models/llama/convert_llama_weights_to_hf.py", line 120, in write_model tokenizer = tokenizer_class(tokenizer_path) File "/opt/conda/lib/python3.10/site-packages/transformers/models/llama/tokenization_llama_fast.py", line 133, in init super().init( File "/opt/conda/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 117, in init slow_tokenizer = self.slow_tokenizer_class(*args, **kwargs) File "/opt/conda/lib/python3.10/site-packages/transformers/models/llama/tokenization_llama.py", line 184, in init self.sp_model = self.get_spm_processor(kwargs.pop("from_slow", False)) File "/opt/conda/lib/python3.10/site-packages/transformers/models/llama/tokenization_llama.py", line 217, in get_spm_processor model = model_pb2.ModelProto.FromString(sp_model) google.protobuf.message.DecodeError: Error parsing message

Expected behavior

no converting

[CallMeFrozenBanana]

This is a very useful support for me, waiting for update

[calmitchell617]

Hi, I am having this same issue. However, the code in question is in the Huggingface Transformers lib, not in llama-recipes. I've opened an issue there.

[ArthurZucker]

Hey ! https://github.com/huggingface/transformers/pull/30334 was opened 3 days ago for this 😉

[HamidShojanazeri]

@yuri-son can you also pls pass --llama_version 3 also --model_Size 8B using the script @ArthurZucker shared in the PR.

In general you wont need to use the conversion script can access HF converted checkpoints from here, https://huggingface.co/meta-llama

[HamidShojanazeri]

closing this issue but feel free to re-open if seeing similar issues.