Exporting Llama3's tokenizer

[vifi2021]

Hello,

I am following https://github.com/pytorch/executorch/blob/main/examples/models/llama2/README.md#option-c-download-and-export-llama3-8b-model to make Llama3-8B-instruct to run on an S21 Ultra. But seems like the examples.models.llama2.tokenizer.tokenizer cannot process llama3's tokenizer.model.

Has anyone run into this issue?

[iseeyuan]

@larryliu0820 , could you help look at this?

[larryliu0820]

Hello you don't need to process tokenizer.model you can just feed it into this step: https://github.com/pytorch/executorch/blob/main/examples/models/llama2/README.md#step-4-run-on-your-computer-to-validate

[vifi2021]

Thanks for the fast response. But I am not running it on my computer. I am running it on an Android phone. In https://github.com/pytorch/executorch/blob/main/examples/models/llama2/README.md#step-5-run-benchmark-on-android-phone step 2.2, it requires to upload model, tokenizer to the phone:

adb push <model.pte> /data/local/tmp/llama/
adb push <tokenizer.bin> /data/local/tmp/llama/

Looks like we still need tokenizer.bin?

[jonatananselmo]

Just do:

adb push <tokenizer.model> /data/local/tmp/llama/

And use <tokenizer.model> wherever you need to specify the tokenizer.

[JacobSzwejbka]

@kirklandsign Since theres some android question can u take a look

[mergennachin]

@vifi2021

In Step 4, we have this "For Llama3, you can pass the original tokenizer.model (without converting to .bin file)."

It also applies to subsequent steps.

@kirklandsign

In our readme file, for Step 5, 2.2 and 2.3, we make that clear.

[kirklandsign]

We updated our setup and readme recently. Please check out the latest one.