EOS_TOKEN = tokenizer.eos_token not working for unsloth/llama-3-8b-Instruct-bnb-4bit

[burgerbee]

Looks like we are getting the wrong EOS_TOKEN and endless generation for the Llama 3 Instruct variant.

[danielhanchen]

@burgerbee The EOS token seems correct no?

Also the generation config:

I did update the models, so you might have to re download them

[burgerbee]

The example code in the notebook https://colab.research.google.com/drive/135ced7oHytdxu3N2DNe1Z0kqjyYIkDXp?usp=sharing works with unsloth/llama-3-8b-bnb-4bit but changing to unsloth/llama-3-8b-Instruct-bnb-4bit gives endless generation.

Will delete model, pip install --upgrade --force-reinstall --no-cache-dir and try again.

[Tonic3]

this is what mstudio team was think it is

Because LM studio defaults to the proper template for that. So in this case, manually select the Llama 3 preset ... if it's still broken when using the right template, the fine tune could be messed up in that it removed the stop Tokens (<|eot_id|>)

[danielhanchen]

Hmm interesting on LM Studio