### System Info

[Karliz24]

System Info

Hello developer, The Llama-3 model was released today.

I want to convert this model to a hf model, but when I follow the readme, the following issue occurs. File "/workspace/00_develop/llama/llama-recipes/./transformers/src/transformers/models/llama/convert_llama_weights_to_hf.py", line 339, in <module> main() File "/workspace/00_develop/llama/llama-recipes/./transformers/src/transformers/models/llama/convert_llama_weights_to_hf.py", line 326, in main write_model( File "/workspace/00_develop/llama/llama-recipes/./transformers/src/transformers/models/llama/convert_llama_weights_to_hf.py", line 120, in write_model tokenizer = tokenizer_class(tokenizer_path) File "/opt/conda/lib/python3.10/site-packages/transformers/models/llama/tokenization_llama_fast.py", line 133, in __init__ super().__init__( File "/opt/conda/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 117, in __init__ slow_tokenizer = self.slow_tokenizer_class(*args, **kwargs) File "/opt/conda/lib/python3.10/site-packages/transformers/models/llama/tokenization_llama.py", line 184, in __init__ self.sp_model = self.get_spm_processor(kwargs.pop("from_slow", False)) File "/opt/conda/lib/python3.10/site-packages/transformers/models/llama/tokenization_llama.py", line 217, in get_spm_processor model = model_pb2.ModelProto.FromString(sp_model) google.protobuf.message.DecodeError: Error parsing message

I would really appreciate it if you could give me some guidance on how to solve this problem. Please help me. thank you!!!

Information

• [X] The official example scripts
• [ ] My own modified scripts

🐛 Describe the bug

'python ./transformers/src/transformers/models/llama/convert_llama_weights_to_hf.py
--input_dir /workspace/00_develop/llama/llama3/llama3-model/Meta-Llama-3-8B
--model_size 7B
--output_dir /workspace/00_develop/llama/llama3/hf-llama3-model/hf_Meta_Llama_3_8B'

Error logs

raceback (most recent call last): File "/workspace/00_develop/llama/llama-recipes/./transformers/src/transformers/models/llama/convert_llama_weights_to_hf.py", line 339, in main() File "/workspace/00_develop/llama/llama-recipes/./transformers/src/transformers/models/llama/convert_llama_weights_to_hf.py", line 326, in main write_model( File "/workspace/00_develop/llama/llama-recipes/./transformers/src/transformers/models/llama/convert_llama_weights_to_hf.py", line 120, in write_model tokenizer = tokenizer_class(tokenizer_path) File "/opt/conda/lib/python3.10/site-packages/transformers/models/llama/tokenization_llama_fast.py", line 133, in init super().init( File "/opt/conda/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 117, in init slow_tokenizer = self.slow_tokenizer_class(*args, **kwargs) File "/opt/conda/lib/python3.10/site-packages/transformers/models/llama/tokenization_llama.py", line 184, in init self.sp_model = self.get_spm_processor(kwargs.pop("from_slow", False)) File "/opt/conda/lib/python3.10/site-packages/transformers/models/llama/tokenization_llama.py", line 217, in get_spm_processor model = model_pb2.ModelProto.FromString(sp_model) google.protobuf.message.DecodeError: Error parsing message

Expected behavior

no converting

Publicación original de @yuri-son en https://github.com/meta-llama/llama-recipes/issues/445

[calmitchell617]

Copying and pasting my response to your original issue:

Hi, I am having this same issue. However, the code in question is in the Huggingface Transformers lib, not in llama-recipes. I've opened an issue there.

[Karliz24]

Información del sistema

Hola desarrollador, hoy se lanzó el modelo Llama-3.

Quiero convertir este modelo a un modelo hf, pero cuando sigo el archivo Léame, ocurre el siguiente problema. File "/workspace/00_develop/llama/llama-recipes/./transformers/src/transformers/models/llama/convert_llama_weights_to_hf.py", line 339, in <module> main() File "/workspace/00_develop/llama/llama-recipes/./transformers/src/transformers/models/llama/convert_llama_weights_to_hf.py", line 326, in main write_model( File "/workspace/00_develop/llama/llama-recipes/./transformers/src/transformers/models/llama/convert_llama_weights_to_hf.py", line 120, in write_model tokenizer = tokenizer_class(tokenizer_path) File "/opt/conda/lib/python3.10/site-packages/transformers/models/llama/tokenization_llama_fast.py", line 133, in __init__ super().__init__( File "/opt/conda/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 117, in __init__ slow_tokenizer = self.slow_tokenizer_class(*args, **kwargs) File "/opt/conda/lib/python3.10/site-packages/transformers/models/llama/tokenization_llama.py", line 184, in __init__ self.sp_model = self.get_spm_processor(kwargs.pop("from_slow", False)) File "/opt/conda/lib/python3.10/site-packages/transformers/models/llama/tokenization_llama.py", line 217, in get_spm_processor model = model_pb2.ModelProto.FromString(sp_model) google.protobuf.message.DecodeError: Error parsing message

Agradecería mucho que me pudieran orientar sobre cómo solucionar este problema. Por favor ayúdenme. ¡¡¡Gracias!!!

Información

• [x] Los scripts de ejemplo oficiales
• [ ] Mis propios scripts modificados

🐛 Describe el error

'python ./transformers/src/transformers/models/llama/convert_llama_weights_to_hf.py --directorio_de_entrada /espacio_de_trabajo/00_desarrollo/llama/llama3/llama3-model/Meta-Llama-3-8B --directorio_de_modelo 7B --directorio_de_salida /espacio_de_trabajo/00_desarrollo/llama/llama3/hf-llama3-model/hf_Meta_Llama_3_8B'

Registros de errores

raceback (última llamada reciente): Archivo "/workspace/00_develop/llama/llama-recipes/./transformers/src/transformers/models/llama/convert_llama_weights_to_hf.py", línea 339, en main() Archivo "/workspace/00_develop/llama/llama-recipes/./transformers/src/transformers/models/llama/convert_llama_weights_to_hf.py", línea 326, en main write_model( Archivo "/workspace/00_develop/llama/llama-recipes/./transformers/src/transformers/models/llama/convert_llama_weights_to_hf.py", línea 120, en write_model tokenizer = tokenizer_class(tokenizer_path) Archivo "/opt/conda/lib/python3.10/site-packages/transformers/models/llama/tokenization_llama_fast.py", línea 133, en init super(). init ( Archivo "/opt/conda/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", línea 117, en init slow_tokenizer = self.slow_tokenizer_class(*args, **kwargs) Archivo "/opt/conda/lib/python3.10/site-packages/transformers/models/llama/tokenization_llama.py", línea 184, en init self.sp_model = self.get_spm_processor(kwargs.pop("from_slow", False)) Archivo "/opt/conda/lib/python3.10/site-packages/transformers/models/llama/tokenization_llama.py", línea 217, en get_spm_processor model = model_pb2.ModelProto.FromString(sp_model) google.protobuf.message.DecodeError: Error al analizar el mensaje

Comportamiento esperado

Sin conversión

Publicación original de@yuri-sonen meta-llama/llama-recipes#445