Output deterioration with model.eval() (unsloth/Meta-Llama-3.1-8B)

[yoakiyama]

Hi Unsloth team,

Thanks for your awesome work! I ran a finetuning experiment on Friday and tried to run a similar experiment on Sunday but ran into errors (something to do with the unsloth code setting the tokenizer to a boolean rather than a tokenizer object), which I suspect arose from changes in the huggingface config files. I installed the latest version of Unsloth, but now see that the model outputs under model.eval() mode are entirely corrupted, while the model.train() outputs appear to be fine. When I inspect the logit argmaxes, I see that they are entirely token 198, besides the BOS token and |end_of_text| tokens. I noticed that there were several pip package updates in the past few days. Would any of these effected Llama3 eval mode? Thank you very much in advance!